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Abstract 

This  paper  proposes  a  mathematical  model  and  formalism  to  study  coded  exposure 
(flutter  shutter)  cameras.  The  model  includes  the  Poisson  photon  (shot)  noise  as  well  as 
any  additive  (readout)  noise  of  finite  variance.  This  is  an  improvement  compared  to  our 
previous  work  that  only  considered  the  Poisson  noise.  In  addition,  closed  formulae  for  the 
Mean  Square  Error  and  Signal  to  Noise  Ratio  of  the  coded  exposure  method  are  given. 

These  formulae  take  into  account  for  the  whole  imaging  chain,  i.e.,  the  Poisson  photon 
(shot)  noise,  any  readout  noise  of  finite  variance  as  well  as  the  deconvolution  and  are  valid 
for  any  exposure  code.  In  addition,  we  give  an  explicit  formula  that  gives  an  absolute  upper 
bound  for  the  gain  of  any  coded  exposure  cameras  in  function  of  the  temporal  sampling 
of  the  exposure  code.  The  gain  is  to  be  understood  in  terms  of  Mean  Square  Error  (or 
equivalently  in  terms  of  Signal  to  Noise  Ratio),  with  respect  to  a  snapshot. 

Keywords:  Coded  exposure,  computational  photography,  flutter  shutter,  motion  blur,  mean 
square  error  (MSE),  signal  to  noise  ratio  (SNR). 

1  Introduction 


Since  the  founding  papers  [2,  3,  4,  37,  38,  39]  of  Agrawal,  Raskar  et  ah  coded  exposure 
(flutter  shutter)  method  has  received  at  lot  follow  ups  [1,  11,  12,  14,  16,  17,  18,  19,  27,  28,  29, 
30,  31,  32,  33,  34,  35,  36,  40,  42,  43,  46,  47,  48,  50,  51,  52,  53,  54].  In  a  nutshell,  the  authors 
proposed  to  open  and  close  the  camera  shutter,  according  to  a  sequence  called  “code” ,  during 
the  exposure  time.  By  this  clever  exposure  technique,  the  coded  exposure  method  permits  one 
to  arbitrarily  increase  the  exposure  time  when  photographing  (flat)  scenes  moving  at  a  constant 
velocity.  Note  that  with  a  coded  exposure  method  only  one  picture  is  stored/transmitted.  A 
rich  body  of  empirical  results  that  suggest  that  the  coded  exposure  method  allows  for  a  gain 
in  terms  of  Mean  Square  Error  (MSE)  or  Signal  to  Noise  Ratio  (SNR)  compared  to  a  classic 
camera,  i.e.,  a  snapshot.  Therefore,  the  coded  exposure  method  seems  to  be  a  magic  tool  that 
should  equip  all  cameras.  However,  to  the  best  of  our  knowledge,  little  is  known  on  the  coded 
exposure  method  from  a  rigorous  mathematical  point  of  view.  Hence,  it  seems  crucial  for  the 
applications  to  build  a  theory  able  to  shed  some  light  on  this  coded  exposure  method.  Eor 
instance,  to  the  best  of  our  knowledge,  little  is  know  on  the  gain,  in  terms  of  MSE  or  SNR,  of 
this  coded  exposure  method  compared  to  a  standard  (snapshot)  camera.  This  papers  proposes 
a  mathematical  model  of  photon  acquisition  by  a  light  sensor.  The  model  can  cope  with  any 
additive  readout  noise  of  finite  variance  in  addition  to  the  Poisson  photon  (shot)  noise.  The 
model  is  compatible  with  the  Shannon- Whittaker  framework,  assumes  that  the  relative  camera 
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scene  velocity  is  constant  and  known,  that  the  sensor  does  not  saturate,  that  the  quantization 
effects  can  be  neglected,  that  the  readout  noise  has  finite  variance  and  that  the  coded  exposure 
method  allows  for  an  invertible  transformation  among  the  class  of  band  limited  functions  (this 
means  that  the  observed  image  can  be  deblurred  using  a  filter).  Note  that  with  this  model  the 
image  has  a  structure:  the  image  is  assumed  to  be  band  limited. 

To  be  thorough,  a  mathematical  analysis  of  a  camera  requires  to  go  rigorously  from  the 
continuous  observed  scene  to  the  discrete  samples  of  the  final  restored  image.  This  is  needed  to 
mathematically  analyze  the  whole  image  chain:  from  the  photon  emission  to  the  final  restored 
image  via  the  observed  discrete  samples  measured  by  the  camera.  To  the  best  of  our  knowledge, 
the  coded  exposure  method  is  very  useful  for  moving  scenes.  Consequently,  we  need  a  formalism 
capable  of  dealing  with  moving  scenes.  Since  the  observed  scene  moves  continuously  with  respect 
to  the  time  we  adopt  a  continuous  point  of  view.  This  means  that  we  shall  model  the  observed 
scene  as  a  function  s.  Loosely  speaking,  s{x)  give  the  light  intensity  at  a  spatial  position  x. 
(By  opposition,  a  discrete  formalism  would  model  the  observed  scene  as  a  vector  of  R”  but 
requires  a  more  restrictive  assumption,  see  below.)  We  shall  rely  on  the  Shannon- Whittaker 
framework  (see,  e.g,  [15])  to  perform  the  mathematical  analysis  of  sampling-related  questions. 
This  framework  requires  the  structure  of  band  limited  (with  a  cut  off  frequency)  signals  or 
images  and  will  allow  us  to  perform  a  rigorous  mathematical  analysis  of  the  coded  exposure 
method.  Recall  that  a  discrete  formalism  would  model  the  observed  scene  as  a  vector  of  R” 
and  the  convolution  would  use  Toepliz  matrices.  Therefore,  the  scene  would  be  assumed  to  be 
periodic  and  also  band  limited  for  sampling  purposes.  Note  that  the  continuous  formalism  that 
we  shall  develop  in  this  paper  does  not  require  to  assume  that  the  observed  scene  s  is  periodic. 
(This  is  needed  because,  to  the  best  of  our  knowledge,  most  natural  scenes  are  not  periodic.) 
However,  the  adaptation  of  the  formalism  that  we  shall  develop  in  this  paper  to  periodic  band 
limited  scene  is  straightforward  if  needed  for  some  application. 

Our  first  goal  is  to  provide  closed  mathematical  formulae  that  give  the  MSE  and  SNR  of 
images  obtained  by  a  coded  exposure  camera.  Therefore,  we  shall  start  by  carefully  model  the 
photon  acquisition  by  a  light  sensor  then  deduce  a  mathematical  model  of  the  coded  exposure 
method.  The  mathematical  model  of  camera  that  we  shall  develop  in  this  paper  has  not,  to  the 
best  of  our  knowledge,  been  developed  in  the  existing  literature  on  the  coded  exposure  method. 
Indeed,  the  model  we  shall  develop  in  this  paper  is  able  to  cope  with  the  Poisson  photon  (shot) 
noise  in  addition  to  any  additive  (sensor  readout)  noise  of  finite  variance  and  does  not  require 
to  assume  that  the  observed  scene  is  periodic.  For  example,  the  model  developed  in  [48]  does 
not  consider  any  additive  (sensor  readout)  noise.  The  formulae  that  give  the  MSE  and  SNR  of 
the  final  crisp  image 

1.  Assumes  the  Shannon- Whittaker  framework  that  i)  requires  band  limited  (with  a  fre¬ 
quency  cut  off)  images,  and  that  ii)  the  pixel  size  is  designed  according  to  the  Shannon- 
Whittaker  theory. 

2.  Assumes  that  the  relative  camera  scene  velocity  is  constant  and  known. 

3.  Assumes  that  the  sensor  does  not  saturate. 

4.  Assumes  that  there  is  no  quantization. 

5.  Assumes  that  the  additive  (sensor  readout)  noise  has  zero  mean  and  finite  variance. 

6.  Assumes  that  the  coded  exposure  allows  for  an  invertible  transformation  among  the  class 
of  band  limited  functions.  (This  means  that  the  observed  image  can  be  deblurred  using  a 
filter.) 
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7.  Neglects  the  boundaries  effects  for  the  deconvolution.  (This  is  in  favor  of  the  coded 
exposure  because  the  inverse  filter  of  a  coded  exposure  camera  has  larger  support  than 
the  inverse  filter  of  a  snapshot.) 

To  the  best  of  our  knowledge,  a  sensor  readout  (additive)  noise  has  zero  mean.  However,  with 
our  formalism,  the  adaptation  to  non  zero  mean  additive  (sensor  readout)  noise  is  straightfor¬ 
ward  if  needed  for  some  application. 

This  paper  proposes  a  mathematical  model  of  coded  exposure  cameras  that  includes  any 
zero  mean  additive  (sensor  readout)  noise  of  finite  variance  in  addition  to  the  Poisson  photon 
(shot)  noise.  This  model  allows  us  to  prove  a  theorem  that  gives  closed  formulae  for  the  MSE 
and  SNR  of  coded  exposure  cameras  (see  theorem  3.8  page  23).  To  the  best  of  our  knowledge, 
the  formula  that  gives  the  SNR  of  the  coded  exposure  method,  cannot  be  found  in  the  existing 
literature  on  the  coded  exposure  method.  We  also  give  an  upper  bound  for  the  gain  of  the 
coded  exposure  method,  in  terms  of  MSE  and  SNR  (see  corollary  4.2  page  31),  with  respect  to 
a  snapshot  in  function  of  the  temporal  frequency  sampling  of  the  exposure  code.  To  the  best 
of  our  knowledge,  the  result  stated  in  corollary  4.2  cannot  be  found  in  the  existing  literature. 
The  bound  is  applicable,  to  the  best  of  our  knowledge,  to  every  existing  exposure  code  of  the 
literature. 

The  upper  bound  of  corollary  4.2  page  31  is  illustrated  on  figure  2  page  31.  In  addition, 
table  1  page  33  provides  numerical  experiments  illustrating  these  results. 

The  paper  is  organized  as  follows.  Section  2  gives  a  mathematical  model  classic  cameras. 
This  mathematical  model  is  extended  in  section  3  to  model  coded  exposure  cameras.  Section  4 
gives  an  upper  bound  for  the  gain  of  the  coded  exposure  method,  in  terms  of  MSE  and  SNR 
with  respect  to  a  snapshot,  in  function  of  the  temporal  sampling  of  the  code.  The  annexes  A- 
G  contain  several  proofs  of  propositions  that  are  used  throughout  this  paper.  A  glossary  of 
notations  is  in  annex  H  page  40.  (In  the  sequel  latin  numerals  refer  to  the  glossary  of  notations 
page  40.) 

2  A  mathematical  model  of  classic  cameras 


The  goal  of  this  section  is  to  provide  a  mathematical  model  of  the  photon  acquisition  by  a 
light  sensor  and  the  formalism  that  we  shall  use  in  the  sequel. 

As  usual  in  the  coded  exposure  literature  [3,  4,  5,  12,  33,  34,  48,  54]  and  for  the  sake  of 
the  clarity  we  shall  formalize  the  coded  exposure  method  using  a  one  dimensional  framework. 
In  other  words,  the  sensor  array  and  the  observed  image  are  assumed  be  one  dimensional. 
This  one  dimensional  framework  is  no  limitation.  Indeed,  as  we  have  seen,  we  assume  that 
the  image  acquisition  obeys  the  Shannon- Whittaker  sampling  theory.  This  means  that  the 
frequency  cut  off  is  compatible  with  the  image  grid  sampling.  The  extension  to  any  two  di¬ 
mensional  grid  (and  two  dimensional  images)  is  straightforward.  The  sketch  of  the  proof  is  in 
annex  A  page  33.  Therefore,  the  one  dimensional  framework  that  we  shall  consider  is  no  lim¬ 
itation  for  the  scope  of  this  paper  that  proposes  a  mathematical  analysis  of  coded  exposure 
cameras.  A  fortiori,  the  calculations  of  MSE  and  SNR  that  we  shall  propose  in  this  paper  re¬ 
main  valid  for  two  dimensional  images.  In  addition,  we  will  assume  that  the  motion  blur  kernel 
is  known,  i.e.,  the  relative  camera-scene  velocity  vector  and  the  exposure  code  (or  function)  are 
known  (this  kernel  is  called  “PSE  motion”  in,  e.g.,  [2]). 

We  now  turn  to  the  mathematical  model  of  photons  acquisition  by  a  light  sensor. 
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2.1  A  mathematical  model  of  photons  acquisition  by  a  light  sensor  that 
includes  any  additive  (sensor  readout)  noise  of  finite  variance  in  addition 
to  the  Poisson  photon  (shot)  noise 

The  goal  of  this  subsection  is  to  give  a  rigorous  mathematical  definition  (see  definition  2.1 
page  8)  of  the  samples  produced  by  a  pixel  sensor  that  observes  a  moving  scene.  This  definition 
of  the  observed  sample  can  cope  with  any  additive  zero  mean  (sensor  readout)  noise  of  finite 
variance  in  addition  to  the  standard  Poisson  photon  (shot)  noise.  Note  that  the  model  developed 
in  [48]  do  not  consider  any  additive  (sensor  readout)  noise.  Therefore,  the  results  of  [48]  do  not 
include  this  more  elaborated  mathematical  model.  In  particular,  the  advantages  of  the  coded 
exposure  method  in  terms  of  MSE,  with  this  more  elaborated  set  up,  are,  to  the  best  of  our 
knowledge,  open  questions.  Note  that,  to  the  best  of  our  knowledge,  the  optimization  of  a  coded 
exposure  camera  is  an  open  question. 

We  consider  a  continuous  formalism  in  order  to  ease  the  transition  from  steady  scenes  to 
scenes  moving  at  an  arbitrary  real  velocity.  Another  advantage  of  this  continuous  formalism  is 
that  it  allows  us  to  avoid  the  implicit  periodic  assumption  of  the  observed  scene  needed  if  one 
uses  Toeplitz  matrices  to  represent  the  convolutions.  (This  is  needed  because,  to  the  best  of 
our  knowledge,  most  natural  scenes  are  not  periodic.) 

We  now  sketch  the  construction  of  our  camera  model.  We  first  consider  the  photon  emission, 
then  include  the  optical  and  sensor  kernels,  then  include  the  effect  of  the  exposure  time  and 
of  the  motion  to  our  model.  The  lasts  two  steps  consists  in  adding  the  Poisson  photon  (shot) 
noise  and  the  additive  (sensor  readout)  noise  to  our  camera  model.  The  camera  model  that  we 
shall  consider  in  this  paper  is  depicted  in  figure  1. 


^  Readout  _  Measurements 
noise 

T  I  I 

Photon  intensity  Optical  Sensor 

emsission  (time  system  array 

evolving) 

Convolution  Convolution 

Figure  1:  Schematic  diagram  of  our  camera  model.  The  observed  scene  emits  light  and  moves  at 
velocity  v.  The  light  undergoes  the  blur  of  the  optical  system  and  is  measured  by  a  pixel  sensor. 
The  pixel  sensor  produces  a  Poisson  random  variable  (shot  noise)  that  is  further  corrupted  by 
an  additive  (sensor  readout)  noise  of  finite  variance  to  produce  the  observed  sample. 

We  assume  that  the  observed  scene  emits  photons  at  a  deterministic  rate  s  defined  by 

s  :  [R  — >  (0,  +oo) 

X  I — >  s(x). 

Here  and  in  the  sequel,  the  variable  x  represents  the  spatial  position.  (We  will  precise  the 
unit  of  X,  i.e.,  the  unit  we  shall  use  to  measure  distances  when  we  introduce  the  pixel  sensor.) 
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Intuitively,  s  represents  the  ideal  crisp  image,  i.e.,  the  image  that  one  would  observed  if  there 
were  no  noise  whatsoever,  no  motion,  with  a  perfect  optical  system  (formally  the  point  spread 
function  is  a  Dirac-mass)  and  the  pixel  sensor  has  an  infinitesimal  area.  In  a  nutshell,  s(x) 
would  be  the  grey  level  of  the  image  at  position  x  in  the  idealistic  case  mentioned  above.  The 
quantity  s(x)  can  also  be  seen  as  the  intensity  of  light  emission  at  x. 

We  now  introduce  the  optical  system  in  our  model.  The  effect  of  the  optical  system  is 
described  by  its  point  spread  function  (PSF)  denoted  g,  and  we  assume  that  g  ^  0.  Formally, 
the  effect  of  the  point  spread  function  is  modeled  by  a  convolution  in  space  (see,  e.g.,  [24, 
equation  7.1  page  171]  see  also,  e.g.  [2,  equation  1,  section  2]).  Therefore,  in  the  noiseless  case, 
if  there  is  no  motion,  the  grey  level  of  the  acquired  image  at  position  x  is,  formally,  described 
by 

(g*s)(x)  (1) 

where  *  denotes  the  convolution  (see  (x)  for  the  definition).  (Recall  that  here  and  in  the  rest 
of  the  text,  Latin  numerals  refer  to  the  formulae  in  the  final  glossary  page  40.)  We  will  give  the 
assumptions  on  g  and  s  so  that  the  quantity  in  (1)  is  well  defined  later  on. 

A  pixel  sensor  can  be  small  but  has  nevertheless  a  positive  area.  Indeed,  a  pixel  sensor 
integrates  the  incoming  light  g  *  s  (the  scene  is  observed  through  the  optical  system)  on  some 
surface  element  of  the  form  [xi,X2]  IR  with  xi  <  X2-  Therefore,  formally,  the  output  of  a 
pixel  sensor  supported  by  [xi,X2],  in  the  noiseless  case  and  without  motion,  is 

rX2 

{g*s){y)dy.  (2) 

Jxi 

In  the  sequel  we  shall  assume  that  all  the  pixel  sensors  of  the  sensor  array  have  the  same  length. 
Mathematically,  we  can  normalize  this  length  so  that  the  all  the  pixel  sensors  of  the  array  have 
unit  length.  This  correspond  to  using  the  pixel  sensor  length  as  unit  to  measure  distances. 
Thus,  this  represents  no  limitation.  Hence,  from  now  on  the  unit  of  x  is  the  pixel  sensor  length. 
By  definition,  with  this  unit,  all  the  pixel  sensor  have  lengths  1.  Therefore,  from  now  on  when 
we  speak  of  a  pixel  sensor  centered  at  x  we  mean  that  the  pixel  sensor  is  supported  on  the 
interval  [x  —  x  -I-  ^]. 

Hence,  from  (2)  we  deduce  that  the  output  of  a  pixel  sensor  supported  on  the  interval 
[x  —  ^,x  -I-  ^],  that  stares  at  the  scene  s  through  the  optical  system  modeled  by  g,  is,  in  the 
noiseless  case  and  without  motion. 


r+5 

^  {g  *  s){y)dy  =  g  *  s){x). 

Ja;-i  V - - / 


(3) 


Consider  the  deterministic  function  formally  defined  hy  u  :=  Ir-iii  *  g  *  s.  The  deterministic 
quantity  u(x)  represents  the  grey  level  of  the  image  at  position  x  if  there  were  no  noise  and  no 
motion.  Indeed,  u  contains  the  kernels  of  the  optical  system  g  and  of  the  sensor.  Note  that  the 
quantity  u(x)  can  also  be  seen  as  an  intensity  of  light  emission  received  by  a  unit  pixel  sensor 
centered  at  x.  With  the  formalism  of,  e.g.,  [2,  equation  1,  section  2]  l|-_i  ij  represents  “hsensor” 
and  g  represents  “hiens”  • 

We  now  introduce  the  exposure  time  in  our  model.  Indeed,  the  sensor  accumulates  the  light 
during  a  time  span  of  the  form  [ti,t2]  IR,  with  ti  <  t2-  We  denote  by  At  the  positive  quantity 
At  :=  t2  —  ti  called  exposure  time.  Thus,  from  (3),  the  output  of  a  pixel  sensor  centered  at  x 
that  integrates  on  the  time  interval  [ti,t2]  is,  in  the  noiseless  case 

rt2  rx+^  rt2 

{g  *  s){y)dydt  =  u{y)dt.  (4) 

Jti  Jtl 
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Note  that  the  quantity  in  (4)  is  the  amount  of  light  measured  by  the  pixel  sensor,  and  it  evolves 
linearly  with  the  exposure  time  At  (=  t2  —  ^i)- 

We  now  extend  the  above  formalism  to  cope  with  moving  scenes.  Without  loss  of  generality 
(w.l.o.g.)  we  assume  that  the  camera  is  steady  while  the  scene  s  moves.  The  coded  exposure 
method  permits  to  deal  with  uniform  motions.  Therefore,  we  assume  that  the  scene  s  moves 
at  a  constant  velocity  n  6  [R  (measured  in  pixel  per  second)  during  the  exposure  time  interval 
[ti,  t2]-  This  means  that  the  scene  evolves  with  respect  to  the  time  as  s(x  —  vt).  Here  and  in  the 
sequel  the  temporal  variable  is  denoted  by  t.  Therefore,  from  (4)  we  deduce  that  the  output  of 
a  pixel  pixel  sensor  centered  at  x  and  integrating  on  the  time  interval  [ti,f2]  is,  in  the  noiseless 
case, 

rt2  rh 

{g  *  s){y  —  vt)dydt  =  u{x  —  vt)dt.  (5) 

Jtl  Jtl 

For  example,  suppose  that  we  take  a  constant  velocity  n  =  1  in  (5).  In  this  case,  the  output  of 
a  pixel  sensor  centered  at  x  is,  in  the  noiseless  case, 


nx+i 

{g  *  s){y  -  t)dydt  = 
•-h 


11]  *  *  s)(a;)  =  *  u){x), 


where  l[a,b]  represents  the  characteristic  function  of  the  interval  [a,  6].  (See  (3)  for  the  last 

equality.)  From  this  simple  example  we  can  qualitatively  describe  where  the  exposure  code  will 

act.  Indeed,  by  a  clever  exposure  technique,  the  coded  exposure  method  will  allow  to  replace 

the  function  l[q,t2]  ™  above  formula  by  a  more  general  class  of  functions  that  does  not  need 

to  be  window  functions.  With  the  formalism  of,  e.g.,  [2,  equation  1,  section  2]  l[q,t2]  represents 

“?)  ■  ” 

'^motion  • 

We  now  extend  our  model  to  cope  with  the  Poisson  photon  (shot)  noise  and  then  will  add 
the  readout  noise.  The  photon  emission  follows  a  Poisson  distribution  see,  e.g.,  [7].  (If  A  is  a 
random  variable  that  follows  a  Poisson  distribution  then  all  the  possible  realization  of  X  are  in 
IN.  In  addition,  the  probability  of  the  event  X  =  k  \s  V{X  =  k)  =  — |i — ,  where  A  >  0  is  the 
intensity  of  the  Poisson  random  variable.)  A  pixel  sensor  is  a  photon  counter.  Indeed,  a  pixel 
sensor  integrates  the  photons  that  are  emitted  by  the  moving  observed  scene  s  on  some  surface 
element  of  the  form  [xi,  X2]  on  the  time  span  [ti,  ^2]  and  produces  a  sample.  This  sample  follows 
a  Poisson  random  variable.  From  (5),  this  means  that  the  sample  produced  by  a  pixel  sensor 
supported  by  [xi,X2]  and  that  integrates  on  the  time  span  [^1,^2]  has  law 


{g  *  s){y  -  vt)dydt 


(6) 


where  the  notation  V{X)  denotes  a  Poisson  random  variable  of  intensity  A.  With  (3)  the  above 
equation  can  be  rewritten  as 


(7) 


Thus,  the  value  of  this  sample  can  be  any  realization  of  this  Poisson  random  variable.  Con¬ 
sequently,  the  probability  that  the  sample  has  value  /c  6  IN  when  observing  the  scene  s  on  the 
time  span  [^1,^2]  with  the  pixel  sensor  centered  at  x  is 


This  quantity  is  nothing  but  the  probability  that  the  pixel  sensor  counts  k  photons  during  the 
time  interval  [^1,^2]- 
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Consider  the  case  where  v  =  0,  ti  =  0,  t2  =  At  in  (7).  If  the  observed  value  ohs{x)  at 
a  position  x  6  [R  follows  V  u{x)dt^  we  have  E(o6s(x))  =  Atu(x).  This  means  that,  in 
expectation,  the  number  of  photon  caught  by  the  pixel  sensor  centered  at  x  increases  linearly 
with  the  exposure  time.  If  we  time  normalize  the  obtained  quantity  and  consider,  formally, 

a  random  variable  lUest(x)  that  follows  ^  °  - -  we  obtain  E(iUest(x))  =  u{x).  This  means 

that  lUest(x)  estimates  u{x)  without  bias.  In  addition,  we  have  var  (lUesi(x))  = 

Consider  the  SNR  on  the  space  interval  [— R,  i?]  given  by 


/  1  rfl 


This  definition 


of  the  SNR  can  be  found  in,  e.g,  [24,  equation  1.39,  p.  42],  [5,  equation  15,  p.  4],  [3,  equation 
1,  p.  2562].  We  have 


^l^R^^est{x))dx 


^l\u{x)dx  _ 


\lm  l^R^^^{^est{x))dx  ^ ^dx 


At 


Therefore,  assuming  that  fj,  the  “mean  signal  level”  [24,  p.  42]  (//  relates  to  zq  in,  e.g.,  [3,  section 

2]) 


1 

IR"*"  9u:=  lim  — -  u(x)dx 

i?->  +  CO  2R 


is  finite  we  can  define  the  SNR  by 


SNR(uesi)  :  = 


limR^+oo  2^  Wr  HTilest{x))dx 
-Y/limR^+00  ^  ^^j^va.r(nest(x))dx 


Thus,  we  have  SNR(iUest)  =  \fJlAi.  For  example,  if  the  mean  photon  emission  fj,  doubles  then  the 
SNR  is  multiplied  by  a  factor  V^.  (And  we  retrieve  the  fundamental  theorem  of  photography.) 
Note  that  if  we  have  no  control  over  the  photon  emission  then  the  only  way  to  increase  the  SNR 
is  to  increase  the  exposure  time  At.  Similarly,  we  define  the  MSE  by 

MSE(iUest)  :=  lim  ^  f  E  (|iUe5t(x)  -  dx, 

R^+oo  ZK  V  / 


whenever  the  limit  exists,  and  we  have  MSE(iUesz)  =  fzAt. 

We  are  now  in  position  to  extend  our  model  to  include  the  additive  (readout)  noise.  Here 
and  in  the  sequel  the  additive  (readout)  noise  of  a  pixel  sensor  centered  at  x  is  modeled  by  a 
zero  mean  real  random  variable  of  finite  variance  denoted  by  r]{x).  Therefore,  from  (6),  the 
output  of  the  pixel  sensor  centered  at  x  that  integrates  the  photons  on  the  time  span 
can  be,  formally,  any  realization  of  the  sum  of  the  random  variables 


^  (l  I  ^^3*s){y -vt)dyd^  +r]{x), 


or  equivalently  (see  (3)), 


+  r?(x). 


(8) 

(9) 


Recall  that  the  deterministic  quantity  u{x)  represents  the  grey  level  of  the  image  at  position 
X  if  there  were  no  noise  and  no  motion  as  it  is  seen  by  a  pixel  sensor  centered  at  x.  Indeed, 
recall  that  u  contains  the  kernels  of  the  optical  system  g  and  of  the  (normalized)  pixel  sensor. 


7 


The  quantity  u{x  —  vt)dt  represents  the  amount  of  light  received  by  a  steady  pixel  sensor 
centered  at  x  that  gathers  the  light  emitted  by  the  observed  scene  moving  at  velocity  v  on  the 
time  interval  Here  and  in  the  sequel,  we  call  “observed  sample”  and  denote  ohs{x)  the 

sum  of  random  variables  formally  defined  in  (9). 

We  now  give  a  mathematical  framework  to  make  precise  the  above  formulae.  We  assume  that 
the  scene  s  s  so  that  the  convolution  in  (3)  page  5  is  well  defined.  We  will  assume  that 

the  PSF  furnishes  a  cut  off  frequency.  This  assumption  is  needed  by  the  Shannon- Whittaker 
sampling  theory.  We  will  assume  that  the  frequency  cut  off  of  g  is  tt,  i.e.,  g  is  [— 7r,7r]  band 
limited.  In  other  words,  g{^)  =  0  for  any  ^  e  [R  such  that  |^|  >  tt,  where,  here  and  in  the 
sequel,  we  denote  by  g  or  J-{g)  the  Fourier  transform  of  g  (see  (xv)  for  the  definition  of  the 
Fourier  transform)  and  (here  and  elsewhere)  ^  6  [R  represents  the  (Fourier)  frequency  coordinate. 
The  choice  of  [— vr,  tt]  in  the  following  definition  is  thoroughly  justified  in  annex  B  page  34. 


Definition  (The  observable  scene  u.) 

We  call  observable  scene  any  non  negative  deterministic  function  u  of  the  form  u  =  lr_i  ii  *g*s. 
Recall  that  the  l[_i  i]  denotes  the  characteristic  function  of  the  interval  [—5,5]  and  is  related 
to  the  normalized  pixel  sensor,  the  PSF  is  g  e  ^(IR)  and  [— vr,  vr]  band  limited  and  s  e 
represents  the  (non  negative)  photon  emission  intensity.  We  have  that  u  6  and  assume 

that  u  satisfies  g  :=  limr/j^+oo  2^  ^^p,u(x)dx  e  IR"''.  In  addition,  we  assume  that  u  :=  u  —  g,  s 
L^{R)  nL2(R). 

Note  that  u  is  the  sum  of  the  constant  g  and  of  u  6  L^(IR).  Thus,  we  have  u  e  5'([R)  (the 
space  of  tempered  distribution).  This  means  that  u  enjoys  a  Fourier  transform  in  5"(IR),  see, 
e.g.,  [6,  p.  173],  see  also  [45,  p.  23].  In  addition,  u  and  u  inherit  the  frequency  cut  off  of  the 
PSF  g.  Therefore,  u  and  u  are  [— tt,  tt]  band  limited.  In  addition  note  that  the  assumption 
u  6  L^([R)  is  w.l.o.g.  Indeed,  since  u  6  L^([R),  from  Riemann-Lebesgue  theorem  (see  e.g.,  [23, 
prop.  2.1]),  we  have  that  u  is  continuous.  In  addition,  since  u  is  [— 7r,7r]  is  band  limited  we 
have  that  u  is  continuous  and  has  compact  support.  We  deduce  that  u  s  L^(IR).  Therefore,  we 
obtain  that  u  s  L^(IR)  w.l.o.g. 

We  can  now  give  a  definition  of  the  observed  sample  at  a  pixel  centered  at  x  that  we  shall 
denote  obs(x). 


Definition  (Observed  sample  of  a  of  pixel  that  includes  any  additive  (sensor  read¬ 
out)  noise  of  finite  variance  in  addition  to  the  Poisson  photon  (shot)  noise.) 

We  assume  that  the  observed  sample  produced  by  a  unit  pixel  sensor  centered  at  x  6  [R  is  cor¬ 
rupted  by  an  additive  noise  ri{x)  that  we  shall  call  readout  noise.  We  assume  that  E{ri{x))  =  0 
and  that  var(ry(x))  =  ar  <  +00.  We  shall  denote  this  observed  sample  by  obs{x),  (from  (9)),  we 
have 


obs{x)  ~  V 


r]{x), 


where  is  the  time  exposure  interval,  the  observable  scene  u  is  defined  by  definition  2.1 

page  8  and  moves  at  velocity  u  6  IR.  The  notation  X  ~  Y  means  that  the  random  variables  X 
and  Y  have  the  same  law. 


In  the  sequel  we  will  need  to  compute  MSEs  as  well  as  SNRs.  Therefore,  we  will  need  to 
compute  expected  values  and  variances  of  the  observed  samples.  Thus,  we  need  to  justify  the 
validity  of  these  operations.  This  is  done  in  annex  C  page  35. 


The  definition  2.1  entails  that  obs{x),  the  observed  sample  of  a  pixel  sensor  centered  at 
position  X,  is  a  measurable  function  (a  random  variable  see,  e.g.,  [44,  p.  168])  for  which  it  is 
mathematically  possible  to  compute,  e.g.,  the  expectation  and  the  variance. 

The  images  produced  by  a  digital  camera  are  discrete.  In  addition,  the  image  obtained  by  a 
coded  exposure  camera  requires  to  undergo  a  deconvolution  to  get  the  final  crisp  image.  Note 
that  the  Wiener  filter  (see,  e.g.,  [22,  p.  205],  [9,  p.  95],  [55,  p.  159]  see  also  [10,  p.  252]  for 
a  definition)  is  not  applicable  in  our  case  since  the  Poisson  noise  is  not  white  additive.  The 
calculation  of  the  adequate  deconvolution  filter  requires  a  continuous  model.  Thus,  we  now  turn 
to  the  sampling  and  interpolation  in  order  to  go  comfortably  from  the  discrete  observations  to 
the  underlying  continuous  image. 

2.2  Sampling  and  interpolation 

This  section  recalls  the  principles  of  the  Shannon- Whittaker  interpolation  that  applies  to, 
e.g.,  images  that  have  the  band  limitedness  structure.  Consider  a  [— 7r,7r]  band  limited  deter¬ 
ministic  function  /  6  T^(IR)  n  L^(IR).  From  the  values  /(n)  for  n  6  Z  the  Shannon- Whittaker 
interpolation  of  /  is 

^  /(n)sinc(x  —  n)  (where  sinc(x)  =  ) 

neZ 

and  the  above  series  converges  uniformly  to  f{x)  for  any  x  6  IR  (see,  e.g.,  [15,  p.  354]). 

We  recall  that  annex  B  page  34  proves  that  it  is  no  loss  to  assume  that  u  is  [— tt,  tt]  band  limited. 
However,  the  sample  obs{x)  defined  in  definition  2.1  page  8  produced  by  the  sensor  is  noisy  as 
it  contains  the  Poisson  photon  (shot)  noise  and  the  additive  sensor  readout  noise.  This  means 
that  obs  is  not  a  deterministic  function.  The  Shannon- Whittaker  theorem  is  usually  applied 
to  deterministic  functions.  Some  generalization  exists  in  the  case  where  the  observed  samples 
are  corrupted  by  an  additive  noise,  see,  e.g.,  [26,  p.  Ill],  or  to  sample  wide  sense  stationary 
stochastic  signals,  see,  e.g.,  [26,  p.  148].  However,  the  Poisson  photon  shot  noise  is  not  additive. 
Therefore,  the  first  generalization  is  not  applicable.  In  addition,  from  definition  2.1  page  8  we 
deduce  that  the  autocorrelation  function  E  (obs{x)obs{y))  is  not  a  function  of  the  variable  x  —  y. 

This  means  that  the  samples  of  a  coded  exposure  camera  cannot  be  seen  as  the  samples  of  a  wide 
sense  stationary  stochastic  process  (see,  e.g.,  [9,  p.  17]  for  the  definition).  Thus,  to  the  best  of 
our  knowledge,  the  existing  generalizations  of  the  Shannon- Whittaker  theorem  are  not  sufficient 
to  treat  the  observed  samples  of  a  coded  exposure  camera  (defined  in  definition  2.1  page  8). 
Consequently,  in  the  sequel,  we  shall  carefully  prove  that,  in  fact,  the  Shannon- Whittaker  theo¬ 
rem  holds  for  the  observed  samples  of  a  coded  exposure  camera.  In  other  words,  we  shall  prove 
that 

“obs(x)  =  ^  obs(n)smc{x  —  n)”  (where  sinc(a;)  =  •)  (10) 

neZ 

is  mathematically  feasible  for  the  obs  defined  in  definition  2.1  page  8. 

Therefore,  in  the  sequel,  we  assume  that  the  observed  samples  are  obtained  from  a  sensor 
array  and  that  the  sensor  array  is  designed  according  to  the  Shannon- Whittaker  sampling 
theory.  Thus,  we  assume  that  the  samples  obs{x)  are  obtained  at  a  unit  rate,  i.e.,  for  x  e  J.. 
Consequently,  we  shall  denote  the  observed  samples  by  obs{n).  This  means  that,  in  the  sequel, 
we  shall  neglect  the  boundaries  effect  due  to  the  deconvolution.  This  is  another  way  to  get  rid 
of  the  boundaries  effects  without  assuming  that  the  observed  scene  is  periodic  as  required  by 
linear  algebra  (with  Toepliz  matrices)  model  based  (see,  e.g.,  [2,  3,  4,  17,  38,  40,  47,  48]). 
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(This  is  needed  because,  to  the  best  of  our  knowledge,  most  natural  scenes  are  not  periodic.) 
Note  that  this  is  in  favor  of  the  coded  exposure  method  since  the  support  of  the  exposure 
function  is  larger  than  the  support  of  the  exposure  function  of  a  snapshot. 

Hereinafter,  we  assume  that  the  sequence  of  random  variables  {rj{n))ne'z  ai'e  mutually  in¬ 
dependent,  identically  distributed  and  independent  from  the  shot  noise,  i.e.,  independent  from 
r  (fi  u(n  —  vt)dt^  where  n  6  Z.  This  independence  assumption  represents  no  limitation  for 
the  model.  Indeed,  a  photon  is  not  caught  twice  by  the  sensor  array.  In  addition,  the  additive 
(sensor  readout)  noise  comes  from  an  inaccurate  reading  of  the  sample  value  that  does  not 
depends  on  the  Poisson  photon  (shot)  noise. 

We  have  defined  the  observed  samples  produced  by  a  light  sensor  in  definition  2.1  page  8. 
This  definition  includes  both  the  Poisson  photon  (shot)  noise  and  an  additive  (readout)  noise  of 
finite  variance.  We  now  turn  to  the  mathematical  formalization  of  the  coded  exposure  method. 

3  A  mathematical  model  of  coded  exposure  camera  that  in¬ 
cludes  any  additive  (sensor  readout)  noise  of  finite  variance 
in  addition  to  the  Poisson  photon  (shot)  noise 


The  goal  of  this  section  is  to  formalize  the  coded  exposure  method.  In  this  section,  we 
consider  invertible  and  deterministic  “exposure  codes”  and  provides  the  MSE  and  SNR  of  these 
exposure  strategies.  The  study  yields  to  theorem  3.8  page  23. 

The  coded  exposure  (flutter  shutter)  method  permits  to  modulate,  with  respect  to  the  time, 
the  photons  flux  caught  by  the  sensor  array.  Indeed,  the  Agrawal,  Raskar  et  al.  flutter  shutter 
method  [2,  3,  4,  37,  38,  39]  consists  in  opening/closing  the  camera  shutter  on  sub-intervals  of 
the  exposure  time.  In  such  a  situation  the  exposure  funetion  that  controls  when  the  shutter 
is  open  or  closed  is  binary  and  piecewise  constant.  Since  it  is  piecewise  constant  it  is  possible 
to  encode  this  function  using  an  “exposure  code” .  (We  give  a  mathematical  definition  of  these 
objects  page  11.) 

Note  that  neither  the  model  nor  the  results  of  [48]  can  be  used  in  this  paper.  Indeed,  in  [48] 
the  additive  (sensor  readout)  noise  is  neglected.  Therefore,  the  formalism  of  [48]  does  not  hold 
with  the  more  elaborated  set  up  that  we  shall  consider  here.  Indeed,  this  paper  considers  any 
additive  sensor  readout  noise  of  finite  variance  in  addition  to  the  Poisson  photon  (shot)  noise. 

As  we  have  seen,  in  their  seminal  work  [2,  3,  4,  37,  38,  39],  Agrawal,  Raskar  et  al.  propose 
to  use  binary  exposure  codes.  Yet,  mathematically,  we  could  envisage  smoother  exposure  codes 
that  are  not  binary.  Indeed,  with  a  bigger  searching  space  for  the  exposure  code  the  MSE  and 
SNR  can  be  expected  to  be  better  than  with  the  smaller  set  of  binary  codes.  Therefore,  in 
the  sequel,  we  shall  assume  that  the  exposure  codes  have  values  in  [0, 1].  The  value  0  means 
that  the  shutter  is  closed  while  the  value  1  means  the  shutter  is  open  and,  e.g.,  ^  means  that 
half  of  the  photons  are  allowed  to  reach  the  sensor.  We  do  not  consider  the  practical  feasibility 
of  these  non  binary  exposure  codes  as  this  is  out  of  the  scope  of  this  paper  which  proposes  a 
mathematical  framework  and  formulae. 

We  first  formalize  the  fact  that  the  exposure  code  method  modulates  temporally  the  flux  of 
photons  that  are  allowed  to  reach  the  sensor  by  giving  an  adequate  definition  of  a  “exposure 
function”  that,  hereinafter,  we  shall  denote  a.  To  be  precise,  the  gain  a{t)  at  time  t  is  defined 
as  the  proportion  of  photons  that  are  allowed  to  travel  to  the  sensor.  We  then  give  the  formula 
of  the  observed  samples  taking  the  exposure  function  into  account  (see  definition  3  page  11). 


10 


Definition  (Exposure  function,  exposure  code.) 

We  call  exposure  function  any  function  a  of  the  form 

a  :  IR  — >  [0, 1] 

t  ' - ^  Tik^-oo°^k^kAt,(k+l)At){t)- 

We  assume  that  e  [0, 1],  for  any  k,  that  {ak)k  ^  and  that  At  >  0.  The  sequence  {ak)k 

is  called  exposure  code. 

Remark  {a  €  L^(R)  n  ^^(IR)  n  L*(IR)) 

The  above  definition  can  cope  with  finitely  supported  codes,  e.g.,  the  Agrawal  Raskar  et  al. 
code  [38,  p.  5]  and  patent  application  [39].  Since,  c  t'^(Z)  c  f*’(Z),  we  deduce  that  for 

any  exposure  code  we  have  a  e  T^([R)  n  L^(IR)  n  L*’(IR). 


We  have  dehned  the  exposure  function  that  controls  with  respect  to  the  time  the  camera 
shutter.  Recall  that  a(t)  is  nothing  but  the  percentage  of  photons  allowed  to  reach  the  sensor  at 
time  t.  We  are  now  in  position  to  give  the  formula  of  the  observed  samples  of  a  coded  exposure 
camera. 

Let  a  be  an  exposure  function  and  u{x  —  vt)  a  scene  moving  at  a  velocity  u  6  IR.  The  exposure 
function  modulates  with  time  the  incoming  photon  flux.  Therefore,  from  definition  2.1  page  8, 
we  deduce  that  obs{n),  the  observed  sample  at  a  position  n  6  Z,  is  a  random  variable 

obs{n)  ~  V  a{t)u{n  —  vt)dt^  +  ri{n)  ~  V  (n)^  +  r/(n). 

Recall  that  the  random  variables  obs(n)  are  only  obtained  for  n  6  Z. 


Definition  (Observed  samples  of  a  coded  exposure  camera.) 

Let  a  be  an  exposure  function.  We  call  observed  samples  at  position  n  of  the  scene  u  (dehned 
in  dehnition  2.1  page  8)  moving  at  velocity  u  6  IR  the  random  variable 


obs{n)  ~  V 


+  r]{n). 


(11) 


Recall  that  the  random  variables  obs{n)  n  6  Z  are  mutually  independent  (see  page  10).  From 
dehnition  2.1  page  8  we  have  that  u  is  of  the  form  L^(IR)  plus  constant  and  is  band  limited. 
From  dehnition  3  page  11  we  have  that  a  6  L^(IR).  We  obtain  that  the  convolution  in  (11)  is 
well  dehned  everywhere.  In  addition,  note  that  the  pixels  are  read  only  once  as  in,  e.g.,  [2,  3,  4, 
37,  38,  39].  This  means  that  with  a  coded  exposure  camera  only  one  image  is  observed,  stored 
and  transmitted. 


Remark  (The  motion  blur  of  a  standard  camera  is  not  invertible  as  soon  as  its 
support  exceeds  two  pixels.) 

A  standard  camera  can  be  seen  as  a  coded  exposure  camera  if  the  exposure  function  a  is  of 
the  form  a  =  ]l[o,Ai])  where  At  >  0  is  the  exposure  time  measured  in  second(s).  Consider  the 
idealistic  noiseless  case  where,  from  (11),  one  would  observe 

E(o6s(n))  =  ^|^l[o,At]  (-)  (n).  (12) 
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From  definition  2.1  page  8  rt  is  [— vr,  tt]  band  limited.  Therefore,  we  deduce  that  the  convolution 
in  (12)  is  non  invertible  as  soon  as  the  Fourier  transform  of  |:^l[o,At]  (t)  vanishes  on  [— vr,  vr]. 

For  any  ^  6  R,  we  have  T  (i^))  (6  =  ^  (l[o,  At])  i^v).  In  addition,  from  the  definition 

of  the  Fourier  transform  (xv),  for  any  ^  6  IR  we  have 

(l[o,At])  (0  =  Atsinc  e-^.  (13) 

Therefore,  for  any  ^  6  R,  we  have 

^  (^l[0.A.l  (-))  ({)  =  Aisinc 

From  the  definition  (xvii)  of  the  sine  function,  we  deduce  that  the  convolution  in  (12)  is  not 
invertible  as  soon  as  the  blur  support  |u|At  ^  2.  Since,  the  velocity  v  is  measured  in  pixed  per 
second,  and  the  exposure  time  At  is  measured  in  second(s)  we  deduce  that  as  soon  as  the  blur 
support  |u|At  exceeds  two  pixels  the  motion  blur  of  a  standard  camera  is  not  invertible. 


The  observed  samples  of  any  coded  exposure  camera  are  formalized  in  definition  3  page  11.  We 
wish  to  compute  the  MSE  and  SNR  of  a  deconvolved  crisp  image  with  respect  to  the  continuous 
observable  scene  u  defined  in  definition  2.1  page  8.  The  study  yields  to  theorem  3.8  page  23. 

To  compute  the  MSE  and  SNR,  a  continuous  deconvolved  crisp  signal  iiest  must  be  defined 
from  the  observed  samples  obs(n)  observed  for  n  6  Z.  Thus,  our  work-plan  is 


Work-plan: 

Step  1  Justify  the  mathematical  feasibility  of  the  Shannon- Whittaker  interpolation  “o6s(x)  = 
Snez  o6s(n)sinc(x  —  n)”  of  the  observed  samples  obs{n)  (see  definition  3  page  11)  of  a 
coded  exposure  camera.  This  is  done  in  section  3.1.  The  study  yields  to  proposition  3.4 
page  16. 

Step  2  Deduce  the  conditions  on  the  exposure  function  a  (see  definition  3  page  11)  for  the 
existence  of  an  inverse  filter  7  that  deconvolve  the  observed  image  and  gives  back  a  crisp 
image.  This  is  done  in  section  3.2.  The  study  yields  to  proposition  3.7  page  19. 

Step  3  Compute  the  MSE  and  SNR  of  this  deconvolved  crisp  image  with  respect  to  the  con¬ 
tinuous  observable  scene  u  (see  definition  2.1  page  8).  This  is  treated  in  section  3.3.  The 
study  yields  to  theorem  3.8  page  23. 


In  order  to  tackle  the  step  1  of  our  work  plan,  the  following  proposition  and  corollary  will 
be  needed  to  compute  the  MSE  and  SNR  of  the  coded  exposure  method. 


Proposition  3.1.  The  observed  samples  of  the  coded  exposure  method  are  such  that,  for  any 
n  sZ, 

E(o6s(n))  =  (“)  *  ^  (^)  var(o6s(n))  =  (“)  *  ^  '^r-  (1^) 

Proof.  The  proof  is  a  direct  consequence  of  definition  3  page  11.  □ 
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Lemma  3.2.  The  deterministic  function  [R  9  x  («)  *  band  limited, 

uniformly  bounded  and  for  any  x  e  R  we  have 

—a  (  -  )  *  tt  j  (x)  =  ( —a  (-^  *  u  )  (x)  +  /i  f  a(t)dt.  (15) 

hi  yuJ  J  Vhl  )  Jr 

Proof.  See  annex  D  page  35.  □ 

Corollary  3.3.  The  deterministic  function  Z  9  n  var(o6s(n))  is  uniformly  bounded. 

Proof.  The  proof  is  immediate  from  proposition  3.1  and  lemma  3.2.  □ 

We  now  turn  to  the  step  1  of  our  work-plan  that  proves  the  mathematical  feasibility  of 
the  Shannon- Whittaker  interpolation,  formally  given  in  equation  (10)  page  9,  of  the  sequence 
obs{n)  where  n  6  Z  to  a  stochastic  process  obs{x)  where  x  6  IR. 

3.1  Mathematical  feasibility  of  the  Shannon- Whittaker  interpolated  image 

obs{x) 

This  section  tackles  the  step  1  of  our  work-plan.  The  study  yields  to  proposition  3.4  page  16. 
We  give  a  rigorous  mathematical  proof  of  the  mathematical  feasibility  of  the  Shannon- Whittaker 
interpolation  of  the  sample  obs{n)  obtained  for  n  6  Z.  In  other  words,  we  need  to  find  a  suitable 
convergence,  for  any  x  6  IR,  of  the  series  formally  defined  by 

00 

obs{x)  =  ^  obs{n)smc{x  —  n).  (16) 

n=— 00 

Recall  that  we  wish  to  compute  the  MSE  and  SNR  of  the  final  crisp  (deconvolved)  image. 
Therefore,  we  need  closed  formulae  for  the  expectation  and  variance  of  (16).  Thus,  we  need  to 
prove  that  the  series  (16)  converges,  for  any  x  6  IR,  in  quadratic  mean.  We  shall  use  a  Cauchy 
argument  (see,  e.g.,  [41,  thm  6.6.2  p.  194]).  We  now  give  the  construction  details. 

Consider  the 


obs{n)  :=  obs{n)  —  E{obs(n)),  (17) 

for  n  6  Z.  The  proof  is  in  three  steps.  The  step  a)  proves  that,  for  any  x  6  IR,  the  series 

-I- GO 

obs(n)smc{x  —  n) 

n=— 00 

converges  in  quadratic  mean.  The  step  b)  proves  that,  for  any  x  6  IR,  the  series 

-I- GO 

E(o6s(n))sinc(x  —  n) 

n=— 00 

converges  to  a  deterministic  constant.  These  two  steps  entail  that  the  series  in  (16)  converges, 
for  any  x  6  IR,  in  quadratic  mean.  The  step  c)  provides  explicit  formulae  for  the  expectation 
and  variance  of  (16).  Recall  that  these  calculations  will  be  needed  for  the  computation  of  MSE 
and  SNR  of  the  deconvolved  crisp  image.  We  now  turn  to  the  proof  of  step  a). 
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Step  a):  From  (17),  for  any  n  6  Z  we  have 

E(o6s(n))  =  0;  var(o6s(n))  =  E(o6s(n)^)  =  var(o6s(n)).  (18) 

Consider  the  finite  sums  of  independent  random  variables 

N 

0^ (x)  :=  ^  obs(n)smc(x  —  n).  (19) 

-N 

Note  that  hereinafter,  denotes  the  Al-th  term  of  the  sequence  defined  in  (19).  For  any 
N  ^  M  and  any  x  6  IR  we  have  that 

E  ^|d^(x)  —  d^(x)|^^  =  ^  E^^o6s(n)^  'j  sinc^(x  —  n).  (20) 

M<|n|s:iV  ^ 


Therefore,  combining  (18)  and  (20)  we  deduce  that 

E  ^|o^(x)  —  o^(x)|^^  =  ^  var(o6s(n))sinc^(x  —  n),  (21) 

M<\n\^N 

for  any  x  6  E.  From  corollary  3.3  page  13  we  have  that  sup„  var(o6s(n))  <  +oo.  Hence, 
from  (21)  we  have 

E  ^|o^(x)  —  o^(x)|^^  ^  fsup var(o6s(n))  j  ^  sinc^(x  — n),  (22) 

for  any  x  6  E.  The  series  X!n*-oo  sinc^(x  —  n)  converges  for  any  x  6  E.  Consequently,  from  (22), 
for  any  x  6  E  and  e  >  0  we  deduce  that  there  exists  Mq  6  El  such  that  if  Mq  ^  M  ^  N  we  have 


Therefore,  from  the  Cauchy  criterion  (see,  e.g.,  [41,  thm  6.6.2  p.  194])  we  deduce  that,  for  any 
X  6  E,  the  series 

+  GO 

obs(x)  =  ^  o6s(n)sinc(x  —  n) 

n=— 00 

converges  in  quadratic  mean  to  a  limit  that  we  can  therefore  call  obs{x).  This  concludes  step  a). 
We  now  turn  to  step  b)  that  proves  that,  for  any  x  6  E,  the  series  X!n*-oo  E(o6s(n))sinc(x  —  n) 
converges  to  a  deterministic  constant.  Recall  that  is  needed  to  conclude  that  (16)  page  13 
converges  in  quadratic  mean. 


Step  b):  From  proposition  3.1  page  12,  for  any  x  6  E,  we  have 


N 

E(o6s(n))sinc(x  —  n) 

n^-N 


Af  /  1  _  X 

f  —a  *  uj  (n)sinc(x  —  n). 

n=—N  ^  ' 


(24) 


Combining  (15)  page  13  and  (24),  we  have,  for  any  x  6  E, 

N 

sinc(x  —  n). 

n=—N 

(25) 


iv  / 1  \  r  r 

E(o6s(n))sinc(x  —  ''^)  = (  ]— ro  (  -  )  *  u  \  (n)sinc(x  —  n)  +  //  a{t)dt 
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We  first  take  care  of  the  first  term  on  the  right  hand  side  (RHS)  of  (25).  From  definition  2.1 
page  8  we  have  u  e  F^([R)  and  u  is  [— 7r,7r]  band  limited.  From  definition  3  page  11  a  6  L^([R). 
Therefore,  from  Young  inequality  (see,  e.g.,  [25,  p.  525])  we  have  that  (:g)  *  e 
and  is  [— vr,  vr]  band  limited  since  u  is  band  limited  in  this  range.  Hence,  we  deduce  (see,  e.g.,  [15, 
p.  354])  that,  for  any  x  6  IR, 


-j-GO  {  "X 

(  -p-j-a  *u  \  (n)sinc(x  —  n)  = 

>  =  —  cr,  \  T I  ^  / 


(26) 


We  now  take  care  of  the  second  term  on  the  RHS  of  (25).  For  any  x  6  IR,  we  have  Xln*-oo  sinc(x  — 
n)  =  1.  Thus,  combining  (25)  and  (26)  we  have  that,  for  any  x  6  IR, 


+00 

E(o6s(n))sinc(x  —  n) 

n=— 00 


(-)  )  (x) 

I  a(t)dt 

\vJ  ) 

Jr 

(27) 


From  definition  2.1  page  8,  for  any  x  6  IR  we  have  u{x)  =  u{x)+fi.  In  addition,  = 

Hence,  from  (27)  we  obtain  that,  for  any  x  6  IR, 


+00 

E(o6s(n))sinc(x  —  n) 

n=— 00 


This  concludes  step  b). 


(28) 


We  now  combine  the  steps  a)  and  b).  Combining  (16)  and  (17)  page  13,  the  series 

+00  +  00  +  00 

o6s(n)sinc(x  —  n)  =  o6s(n)sinc(x  —  n)  +  E(o6s(n))sinc(x  —  n) 

n=— GO  n=— GO  n=— GO 

converges,  for  any  x  6  R,  in  quadratic  mean  to  a  limit  that  we  can  therefore  call  obs(x).  Indeed, 
from  step  a),  it  is  the  sum  of  the  quadratic  mean  convergent  series  o6s(n)sinc(x  —  n) 

and,  from  (28),  of  the  deterministic  constant  (:g)  *  (x).  Consequently,  we  deduce  that, 

for  any  x  6  IR,  the  series  defined  in  (16)  page  13  converges  in  quadratic  mean.  Thus,  we  call 
obs{x),  for  any  x  6  IR,  this  limit.  We  now  turn  to  step  c)  that  gives  explicit  formulae  for  the 
expectation  and  variance  of  the  quadratic  mean  convergent  series  obs(x)  defined  in  (16). 


Step  c):  The  convergence  in  quadratic  mean  implies  the  convergence  of  the  two  firsts 
moments  (see  [20,  Ex  5.6  (a)-(b),  p.  158]).  Therefore,  for  any  x  6  IR  we  have 


N 


E|  E  o6s(n)sinc(x  —  n)  j  E  j  o6s(n)sinc(x  —  n) 


and 


\n=—N 


N 


var  E  o6s(n)sinc(x  —  n) 


N—^CO 


var  E  o6s(n)sinc(x  —  n) 


(29) 


(30) 


In  addition,  the  linearity  of  the  expectation  and  the  independence  of  the  observed  samples 
obs{n)  imply  that,  for  any  x  6  IR, 


N 


N 


M  E  o6s(n)sinc(x  —  n)  j  =  E(o6s(n))sinc(x  —  n) 


(31) 


Vn=— Af 


n=-N 
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and 


N 


N 


var  E  o6s(n)sinc(x  —  n)  I  =  ^  var(o6s(n))sinc^(a;  —  n). 


\n^—N 


n=-N 


Therefore,  combining  (29)  with  (31)  and  the  definition  (16)  page  13  of  obs(x)  we  have 


N 


^  E (o6s(n)) sine (rc  —  n)  E(o6s(x)), 


(32) 


(33) 


n^-N 


for  any  x  6  E.  Similarly,  combining  (30)  with  (32)  and  the  definition  (16)  page  13  of  ohs{x)  we 
have  that 

N 

var (o6s(n)) sine (x  —  n) — var(o6s(x)),  (34) 

n^-N 

for  any  x  6  R.  Combining  (28)  and  (33)  we  have  that,  for  any  x  6  E, 


E(o6s(x))  =  ^  j  (^)- 


From  proposition  3.1  page  12  and  (34)  we  deduce  that,  for  any  x  6  E, 
N 


n=-N  L 


2,  + 


sinc^(x  —  n)  var(o6s(x)). 


(35) 


(36) 


Since,  for  any  x  6  E,  we  have  X!n*-oo  sinc^(x  —  n)  =  1,  from  (36)  we  deduce  that,  for  any  x  6  E, 


Jn  =  — 00 

+  CO 


var(o6s(x))  =  *u\  (n)sinc^(x  —  n) 


+  al. 


(37) 


In  addition,  from  corollary  3.3  page  13,  we  deduce  that,  for  any  x  6  E,  var(o6s(x))  is  finite. 
Thus,  we  proved 


Proposition  3.4.  (Mathematical  feasibility  of  the  Shannon- Whittaker  interpolation 
of  the  observed  samples  obs{n)  n  e  Z.) 

The  Shannon- Whittaker  interpolation  of  the  observed  samples  obs(n)  n  e  Z  defined  in 
definition  3  page  11  is,  for  any  x  6  E,  the  quadratic  mean  convergent  series 

+00 

obs{x)  =  obs{n)smc(x  —  n). 

n=— 00 

From  (35),  its  expectation  is,  for  any  x  6  E, 

E(o6s(x))  =  (-) 

From  (37),  its  variance  is  finite  and,  for  any  x  6  E, 

+00 

var(o6s(x))  = 

n=— GO 

We  proved  that  the  Shannon- Whittaker  interpolation  is  mathematically  feasible  for  the 
observed  samples  of  a  coded  exposure  camera  defined  in  definition  3  page  11.  We  shall  now 
turn  to  the  definition  of  the  deconvolved  crisp  image  that  we  shall  denote  iiest(x). 


j— j-a  f-')  *  u  )  (n)sinc^(x  —  n) 
m  \vJ 


(Tr^ 
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3.2  Mathematical  feasibility  of  the  the  deconvolved  crisp  image  uiest(a;) 


This  section  treats  the  step  2  of  our  work-plan.  The  study  yields  to  proposition  3.7  page  19. 
Recall  that  the  existence  of  obs(x)  =  X!n*-oo  o^'S(n)sinc(x— n)  is  given  in  proposition  3.4  page  16. 

Formally,  we  wish  to  consider  obs  *  7  where  7  will  be  the  deconvolution  filter  that  inverses 
the  convolution  by  the  exposure  function  (-)  in  proposition  3.4  equation  (38).  Note  that 
we  cannot  recur  to  a  Wiener  filter  to  define  7.  Indeed,  due  to  the  Poisson  photon  (shot)  noise, 
the  noise  in  of  our  observations  obs(n)  defined  in  definition  3  page  11  is  not  white  additive. 
Therefore,  to  the  best  of  our  knowledge,  a  Wiener  filter  is  not  defined  (see,  e.g.,  [22,  p.  205],  [9, 
p.  95],  [55,  p.  159]  see  also  [10,  p.  252]  for  a  definition).  Instead  of  using  a  Wiener  filter  we 
shall  propose  a  filter  designed  so  that  the  restored  crisp  image  lUest  is  unbiased. 

Beside  the  definition  of  the  inverse  filter  7,  we  need  to  prove  the  mathematical  feasibility 
of  “06s  *  7” .  This  is  needed  because  obs  given  in  proposition  3.4  page  16  does  not  belong  to  a 
classic  Lebesgue  space.  In  addition  we  need  to  deduce  the  conditions  on  the  exposure  function 
a  (see  definition  3  page  11)  to  guarantee  that  E(7  *  obs(x))  =  u{x)  for  any  x  6  IR,  i.e.,  the 
conditions  on  a  so  that  its  convolution  can  be  inverted. 

We  assume  that  the  exposure  function  a  6  T^(IR)  n  L^(IR)  n  L*’([R)  defined  in  definition  3 
page  11  satisfies  a{^v)  ^  0  for  any  ^  e  [— 7r,7r].  Under  that  condition  the  convolution  (p^«  (i;))* 
u  is  invertible  because  u  is  [— vr,  vr]  band-limited.  Thus,  provided  ci(^u)  ^  0  for  any  ^  6  [— vr,  vr], 
we  can  consider  the  inverse  filter  (defined  by  its  inverse  Fourier  transform  (xv)) 


7(x)  :=  7-1 


/l[-7r,7r]  (0  \ 

V  HCv)  J 


(x). 


(39) 


This  filter  7  will  be  applied  to  the  Shannon- Whittaker  interpolated  image  obs{x)  with  x  6  IR 
given  in  proposition  3.4  page  16.  Since  we  assume  that  a{^v)  7  0  for  any  ^  e  [— vr,  vr]  we 
have  |d(^x)|  >  0  for  any  ^  6  [— 7r,7r].  Hence,  we  deduce  that  IR  B  ^  bounded, 

has  compact  support  and  therefore  belongs  to  T^(IR)  <b  L^(IR).  From  its  definition  (39)  and 
Riemann-Lebesgue  theorem  (see  e.g.  [23,  prop.  2.1])  we  have  that  7  is  continuous.  In  addition, 
we  have  7  6  L^([R)  as  the  inverse  Fourier  transform  of  a  function  that  belongs  to  L^([R).  Thus, 
we  deduce  that 

(7 

7^(x)  ^  — 2  ^  (40) 

for  some  constant  C  >  0.  Furthermore  from  (39)  we  have  7(^)  =  Therefore,  we  have 


7(0) 


1 

d(0) 


1 

^^a{x)dx' 


(41) 


Before  we  turn  to  the  mathematical  feasibility  of  “obs  *  7” ,  we  first  prove  that  the  inverse  filter  7 
defined  in  (39)  gives  an  unbiased  estimator  of  u  that  is  E(o6s)*7  =  u.  Adhuc,  we  used  the  Poisson 
summation  formula  (see  annex  G  page  39)  to  prove  equalities  like  “X!*=-oo9'(^  —  n)  =  7(0)”. 
The  use  of  the  Poisson  summation  formula  requires  deterministic  functions  that  belongs  to 
L^(E).  Since  7  6  L^(IR)  but  a  ^  T^(IR)  we  cannot  recur  to  the  Poisson  summation  formula  (ix) 
used  elsewhere.  Instead  we  introduce  that  following  lemma 


Lemma  3.5.  (Poisson  like  formula  for  the  inverse  filter  7.J 

For  any  x  6  E,  we  have 


+00 

n=— 00 


1 

5^a(x)dx’ 


(42) 
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for  7  defined  by  (39) . 

Proof.  The  proof  is  in  annex  E  page  35. 


□ 


The  next  lemma  means  that  in  the  idealistic  noiseless  case  where  one  observes  E(o6s(n))  for 
n  6  Z,  the  inverse  filter  defined  in  (39)  page  17  allows  us  to  get  back  the  ideal  crisp  image  u. 
This  lemma  will  be  useful  in  the  noisy  (general)  case  and  to  compute  the  MSE  and  SNR  of  the 
coded  exposure  method. 


Lemma  3.6.  (The  inverse  filter  7  gives  back  an  unbiased  estimator  of  u.) 

Provided  that  d(^u)  0  for  any  ^  6  [— 7r,7r],  we  have 


-j-GO  {  ^  \ 

f  Y~\^  (~)  *  ^ )  —  n)  =  u(x), 

7,=  — on  V  ^  / 


(43) 


for  any  x  6  IR  and  7  defined  by  (39)  page  1 7. 

Proof.  The  proof  is  in  annex  F  page  36.  □ 

The  lemma  3.6  page  18  proves  that,  at  least  in  the  idealistic  noiseless  case,  i.e.,  when  one 
observes  the  noiseless  samples  E(o6s(n))  for  n  6  Z,  the  inverse  filter  7  is  able  to  recover  a  crisp 
image.  However,  in  a  practical  situation  the  observed  samples  are  noisy.  Therefore,  we  now 
turn  to  the  general  noisy  case. 

In  order  to  get  an  unbiased  estimate  of  u,  i.e.,  a  final  crisp  image  we  would  like  to  define 


''vLest  ■=  obs  *  (44) 

In  the  sequel,  using  lemma  3.6  page  18,  we  shall  prove  that  E(iUest(x))  =  u{x)  for  any  x  6  IR. 
(This  means  that  iiest  is  an  unbiased  estimator  of  u.)  However,  the  convolution  in  (44)  is  unde¬ 
fined  because  the  random  function  obs  that  comes  from  the  Shannon- Whittaker  interpolation 
of  the  observed  samples  defined  in  proposition  3.4  page  16  does  not  belong  to  any  Lebesgue 
space.  Therefore,  we  now  justify  the  mathematical  feasibility  of  (44)  that  (formally)  defines  the 
final  crisp  image. 

From  its  definition  (39)  page  17,  we  have  that  7  is  compactly  supported  (on  [— 7r,7r])  and 
bounded.  Therefore,  7  6  L^([R)  n  L^([R).  From  7  6  T^([R)  and  Riemann-Lebesgue  theorem  (see 
e.g.  [23,  prop.  2.1])  we  deduce  that  7  is  bounded.  In  addition,  from  7  e  L^(IR)  we  have  that 
7  6  L^(IR).  Consequently,  7  is  [— 7r,7r]  band-limited  and  C'*’(IR),  bounded,  and  belongs  to  L^([R). 
This  means  that  7  is  a  nice  inverse  filter. 

We  need  to  recur,  again,  to  a  convergence  argument  for  the  definiteness  of  Uest-  Since,  7  is 
[— 7r,7r]  band  limited,  for  any  x  6  IR,  we  have  (sinc(-  —  n)  *  7)  (x)  =  7(x  —  n).  Therefore,  for  any 
X  6  [R,  we  set 


N 

:  =  J]  obs{n)-f{x -n).  (45) 

n=-N 

Note  that  hereinafter,  lu^^(x)  denotes  the  Wth  term  of  the  sequence  defined  in  (45). 

Our  goal  is  to  find  an  adequate  mode  of  convergence,  for  any  x  6  IR,  of  these  finite  sums  to 

00 

iiest(x)  :=  y]  obs{n)-f{x  -  n). 
n=— 00 
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From  (45),  for  any  x  6  IR,  we  have 


N 

H^estix))  =  J]  E{obs{n))-f{x  -n). 

n=-N 

Combining  proposition  3.1  page  12  and  (46),  for  any  x  6  IR,  we  have 

N 

=  J] 

n^-N 

From  lemma  3.6  page  18  we  deduce  that,  for  any  x  6  [R, 

E(TLi^t(a;))  u(x). 


R"  (x)  *  V 


7(x  —  n). 


(46) 


(47) 


(48) 


We  have 


Proposition  3.7.  (Validity /Existence  of  the  crisp  deconvolved  image  Uest-) 

Consider  the  observed  samples  obs(n)  defined  in  definition  3  page  11  and  the  inverse  filter 
7  defined  in  (39)  page  17.  We  have  that,  for  any  x  6  IR,  the  series 

00 

nest(x)  :=  R  obs(n)'y{x  -  n) 
n=— 00 

converges  in  quadratic  mean.  Furthermore,  for  any  x  6  IR,  its  expectation  is 

E(iUest(x))  =  u(x) 


and  its  variance  is  finite  and 


00 

var(iiest(x))  =  R  var(o6s(n))(7(x  -  n))^. 

n=— GO 

This  proposition  means  that  is  an  unbiased  estimator  of  the  observable  scene  u. 


Proof.  The  proof  follows  exactly  the  same  arguments  as  for  the  construction  of  obs{x)  lead¬ 
ing  to  proposition  3.4  page  16.  Indeed,  the  decay  of  7  given  in  (40)  page  17  is  the  same 
as  the  sine.  Therefore,  we  obtain  that,  for  any  x  6  IR,  u/gfix)  =  o6s(n)sinc(x  —  n) 

converges,  in  quadratic  mean,  to  a  limit  that  we  can  therefore  call  lUest(x).  This  mode  of 
convergence  implies  the  convergence  of  the  two  firsts  moments  (see  [20,  Ex  5.6  (a)-(b),  p. 
158]).  Therefore,  we  again  obtain  that,  for  any  x  6  IR,  E(iu^j(x))  E(iUest(x)).  Thus, 

from  (48)  we  deduce  that,  for  any  x  6  IR,  E(TUest(x))  =  u{x).  In  addition,  for  any  x  6  R, 
we  have  var(iu^j(x))  var(iUest(x)).  Furthermore,  the  independence  of  obs{n)  implies 

that,  for  any  x  6  IR,  var(iu^j(x))  =  var(o6s(n))7^(x  —  n).  Thus,  we  have  var(iUest(x))  = 

var(o6s(n))7^(x  —  n)  <  -l-oo  for  any  x  6  IR.  Indeed,  from  corollary  3.3  page  13,  we  have 
sup„  var(o6s(n))  <  -l-oo.  From  its  definition  (39)  page  17  we  have  that  7  is  [— tt,  tt]  band-limited. 
In  addition,  from  (40)  page  17  we  have  that  7^  6  L^(IR).  Hence,  we  deduce  that  7^  is  such  that 
is  supported  on  [— 27r,27r].  Moreover,  from  the  definition  of  7  (39)  page  17  we  have 


72(27r) 


(7  *  7)(27r)  = 


i: 


l[-7r,7r](2vr  -  0 

a{v(,)a{v{27r  —  ^)) 


(49) 
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The  integrand  in  (49)  is  non  zero  only  on  the  zero  Lebesgue  measure  set  {tt}.  Therefore,  we 
have  7^(27r)  =  0.  Similarly,  we  have  7^(— 27r)  =  0.  Thus,  from  the  Poisson  formula  (see  annex  G 
page  39)  we  deduce  that 

+00  +00 

72(27rn)  =  72(0)  =  ^2^^  _  ^ 

n=— 00  n=— GO 

From  corollary  3.3  page  13  we  have  sup„  var(o6s(n))  <  +00.  Hence,  for  any  x  6  IR  we  have 
var(iUest(x))  ^  sup„  var(o6s(n))||7||2,2('|R)  <  +00  which  concludes  the  proof.  □ 

Proposition  3.7  page  19  proves  the  feasibility  of  the  restore  crisp  image  Ues^.  This  concludes 
the  step  2  of  our  work  plan.  We  now  turn  to  the  step  3  of  our  work  plan  that  is  the  calculations 
of  the  MSE  and  SNR  of  lUest  with  respect  to  the  observable  scene  u  (see  definition  2.1  page  8). 

3.3  The  MSE  and  SNR  of  the  restored  crisp  image  lUest 


We  have  defined  the  deconvolved  crisp  image  lUest  in  proposition  3.7  page  19.  We  shall  now 
turn  to  the  step  3  of  our  work-plan  that  provides  closed  formulae  for  the  average  MSE  and  SNR 
of  the  coded  exposure  method.  The  study  yields  to  theorem  3.8  page  23. 

Recall  that,  from  proposition  3.7  page  19,  for  any  x  6  IR,  we  have 

E(iUest(x))  =  u{x).  (51) 


Therefore,  for  any  x  6  IR,  we  have 

E  (^|TUest(a;)  -  =  var(TUest(x)).  (52) 

In  order  to  provide  closed  formulae  for  the  MSE  and  SNR  we  need  to  evaluate  var(iUest(x))  and 
the  limit  of  ^  var(TUest(x))dx  when  R  —>  -too. 

We  now  calculate  var(iUest(x)).  Erom  proposition  3.7  page  19,  for  any  x  6  IR,  we  have 

00 

var(TUest(x))  =  ^  var(o6s(n))(7(x  -  n))^.  (53) 


Combining  (14)  page  12  and  (53),  for  any  x  6  E,  we  have 


vaT{uest{x))  =  ^  *uVn) -tcj^V7(x-n)) 


n=— CO 


(54) 


Therefore,  from  (54)  we  deduce  that,  for  any  x  6  R, 

1 


var 


n=— 00  L 


(Uestix))  =  Yi  (y"  (“)  *^)  +^r  Y  (55) 


Erom  definition  3  page  11,  we  have  that  a  is  non  negative.  Therefore,  combining  (15)  page  13 
and  (55),  for  any  x  6  IR,  we  have 


var  I 


00  r  / 1  \  1 

(lUe^t(x))  =  Y  (  O"  (“)  *^)  (’^)(^(*  “’^))^  +(/^ll«llLl(R)+f^r)  Y  (56) 
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Hence,  combining  (50)  page  20  and  (56),  for  any  x  6  R,  we  have 


var(iuest(x))  =  2  (  T—a  (n)(7(x -n))2  +  (/^II«IIl1(k)  +  ^r)ll7lli2(K).  (57) 

n=— 00  L  \  I  I  / 

Moreover,  from  the  definition  of  7  (39)  page  17  and  Plancherel  identity  (146)  we  have 

1  1 

b\\h(R)  =  ^  (58) 

Therefore,  ||q;||li([R)  =  Thus,  combining  (57)  and  (58)  we  deduce  that,  for  any  x  G  [R, 


00  r  / 1  \ 

var(iue^t(x))=  y  (04“  (v)  *^)  (”)(7(*“”)) 


cr; 


2tt 


Jr  |a(^^ 


Hence,  combining  (52)  and  (59),  for  any  x  6  IR,  we  have 
E . .  '  '  '  ■  ^ 


^  1 2  ^[~7r,7r]  (C^C 

(59) 


n=— 00 


(|iUest(x)  -  u{x)f^=  y  (—a  (-)  *  u)  (n)(j(x  -  n)) 


Ml|a||Ll(R)  +0-r  rl[-7r,7r](0c^^ 


27r 


• 

Jr 


(60) 


We  now  calculate  the  limit  of  ^  var(irest(x))dx  when  R  —>  +oo.  The  function  Rax 
(iT*^^  (-)  *  (n)(j(x  —  n))^  belongs  to  L^(R).  Indeed,  we  have 


ZOO 

n=— GO 

I 


n=— GO  *- 


r  /  1  \  1  /^+oo  r  /  ^  \ 

Z  (  y«(-)*^)(™)(T'(^-^))'  S  ( 

— —  rr^LVm  I  /  J  —  GO  r)  — —  rr^  L  \wl  / 


■'n=—oo  L 


(7(x-n))" 


Hence,  it  follows  from  (61)  and  Fubini  theorem  (see,  e.g.,  [44,  p.  196])  that 

f  S  (A«(-)*^)(^)(7(a;-n))2  dx^Yi  (A«(-)  *^^)  («)  [  -  n))‘ 

n=— cot  \l  I  /  J  i2=— cot  \l  I  /  CO 

For  any  n  s  J.,  we  have 


dx. 

(61) 

dx 
(62) 


p+oo 

(lix-n)fdx  =  \\j\\l2(^^y 

J  — 00 


Therefore,  from  (62)  we  deduce  that 


0+00 
J— 00 


s 


1 

n" 


(-)  *  ^)  {n)i7{x  -  n)  f  dx  ^  ||7|li2(R)  Y  (“)  *  ^)  b) 

/  J  n  =  — oo  V  I  I  / 


•  (63) 


From  definition  2.1  page  8  we  have  u  e  T^(R)  and  u  is  [— tt,  vr]  band  limited.  From  definition  3 
page  11  a  6  L^(R).  Therefore,  from  the  Poisson  summation  formula  (see  annex  G  page  39)  we 
obtain  that 

^  (“)  < +°®-  (6^) 


£  I  (yG)  *")*"> 

=  — oo  \  I  I  / 


n“ 

m 
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Indeed,  from  Young  inequality  (see,  e.g.,  [25,  p.  525])  we  have  that  (;[;)  *  ft j  e  -L^(IR).  In 
addition,  from  (40)  page  17  7  6  L^(IR).  Thus,  combining  (63)  and  (64)  we  deduce  that 


dx  <  +00. 


(65) 


From  (65),  we  deduce  that 


1 


R"  (-)  *  -  n))^ 


(66) 


For  any  R  >  0,  from  (59)  we  have  that 


rR  ^  /  rR 

— -  vai(iaest(x))dx  =  — -  V 

2R]_n  ^  2R\]_j,^^_ 


n=— 00  •- 


1 

n“ 
m  w 


27r 


(-)  *uj  {n){'y{x-n)f 


dx 


Combining  (66)  and  (67)-(68),  we  obtain 


1 

—  var(TUest(x))dx 

2R  J-R 


R^OO  I^II«IIl1(IR)  +  Cf; 


27r 


f  l-fC  M2^[-7r,7r](0^^- 

Jr  |a(^^')r 


Hence,  combining  (52)  page  20  and  (69),  we  deduce  that,  for  any  x  6  IR, 
^  E  (|o„,(x)  -  «(x)|  )  dx - . - 


Thus,  we  proved  the  following  theorem 


(67) 

(68) 


(69) 


(70) 
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Theorem  3.8.  (MSE  and  SNR  of  the  coded  exposure  method.) 

Consider  a  scene  u{x  —  vt)  of  mean  p,  (see  definition  2.1  page  8)  that  moves  at  velocity  v  and 
let  the  (finite)  variance  of  the  additive  (readout)  noise.  From  (70)  the  MSE  of  the  final  crisp 
image  iiest  defined  in  proposition  3.7  page  19  satisfies 


:=  ^lim^  ^  J  e  (|e„,(x)  -  „(i)|2)  dt  =  ^  (71) 


From  (51)  and  (69)  the  SNR  of  a  coded  exposure  camera  is 

lim/j^+oo  ^  E  (lUest(x))  dx 


SNRpy^i^fioi)  . — 


h 


\l  2n  J— TT  \n{iv)\ 


(72) 


We  now  connect  the  formulae  in  theorem  3.8  with  the  existing  literature  on  the  coded 
exposure  method.  We  have  that  the  mean  photon  emission  p  relates  to  io  in,  e.g.,  [3,  section 
2].  In  addition,  from  (72),  we  have  that  for  fixed  exposure  function  a  and  additive  (readout) 
the  SNR  evolves  proportionally  to  /  ^  ^  with  the  mean  photon  emission 


noise  variance  az 


i+- 


p.  In  particular,  from  (72),  if  =  0  and  for  a  fixed  a  we  deduce  that  the  SNR  evolves 
proportionally  to  ..Jp  and  we  retrieve  the  fundamental  theorem  of  photography.  We  now  turn 
to  the  optimization  of  the  coded  exposure  method.  We  shall  optimize  the  coded  exposure 
method  for  given  observed  scene  and  sensor.  This  means  that  the  mean  photon  emission  p  and 
additive  (readout)  noise  variance  a(:  are  fixed.  Under  these  conditions,  from  theorem  3.8  we 
deduce  that  the  optimization  of  the  coded  exposure  method  boils  down  to  finding  an  exposure 
function  a  that  minimizes  the  MSE  given  by  formula  (71)  or,  equivalently,  of  that  maximizes  the 
SNR  given  by  formula  (72).  Notice  that,  to  the  best  of  our  knowledge,  this  is  the  optimization 
framework  considered  in  the  literature  on  the  coded  exposure  method  [1,  3,  2,  5,  17,  29,  27,  34]. 

Recall  that  it  is  equivalent  to  minimize  (71)  or  to  maximize  (72)  with  respect  to  the  exposure 
function  a.  Therefore,  in  the  sequel  we  choose  w.l.o.g.  to  use  formula  (71)  and  to  evaluate  the 
performance  of  the  coded  exposure  method  in  terms  of  MSE.  The  calculation  for  the  SNR  can 
be  immediately  deduced.  We  now  summarize  the  construction  of  section  3. 


To  sum  up,  in  section  3  we  have  defined  the  observed  samples  obs{n)  of  a  coded  exposure 
camera  (see  definition  3  page  11).  We  have  defined  the  Shannon- Whittaker  interpolated  image 
obs{x)  for  any  x  6  IR  (see  proposition  3.4  page  16)  of  the  stochastic  observed  samples  obs{n) 
that  are  only  obtained  for  n  6  Z.  We  have  defined  the  inverse  filter  and  proved  rigorously  the 
existence  of  lUest  i.e.,  the  final  crisp  image.  The  final  crisp  TUest(x)  is  designed  to  ensure  that 
lUest(x)  is  equal  to  the  observable  scene  u{x)  in  expectation  i.e.,  E(iUest(-))  =  In  addition, 
this  final  crisp  image  lUest  is  obtained  from  the  observed  stochastic  samples  obs{n)  (see  propo¬ 
sition  3.7  page  19)  knowing  the  exposure  function  a  and  the  relative  camera  scene  velocity 
v.  Theorem  3.8  gives  the  MSE  and  SNR  of  the  coded  exposure  method  as  a  function  of  the 
exposure  function  a  (see  definition  3  page  11),  of  the  relative  camera  scene  velocity  u,  of  the 
mean  photon  emission  p  and  of  the  additive  noise  variance  (see  definition  2.1  page  8). 

As  an  easy  application  of  theorem  3.8,  we  have  the  following  corollary  that  provides  the  MSE 
of  any  invertible  snapshot,  i.e.  that  satisfies  \v\At  <  2  (see  the  discussion  on  page  12)  where  At 
is  the  exposure  time.  This  corollary  will  also  be  needed  to  compare  the  coded  exposure  method 
and  the  snapshot,  in  terms  of  MSE,  in  section  4. 
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Corollary  3.9.  (MSE  of  a  snapshot  with  an  exposure  time  of  At.) 

Consider  a  scene  u(x  —  vt)  of  mean  p,  (see  definition  2.1  page  8)  that  moves  at  velocity  v 
and  let  a‘^  be  the  (finite)  variance  of  the  additive  (readout)  noise.  The  MSE  of  a  snapshot  with 
exposure  time  At  is 


MSEsnapshoti^t) 


Jit  l„  =  1 


pAt  + 


Atsinc 


idi. 


(73) 


Proof.  The  proof  is  immediate  combining  (13)  page  12  and  (71). 


□ 


We  now  turn  to  section  4  that  proposes  a  theoretical  evaluation  of  the  gain,  in  terms  of 
MSE,  of  the  coded  exposure  method,  with  respect  a  snapshot. 


4  An  upper  bound  of  performance  for  coded  exposure  cameras 

This  section  study  the  gain,  in  terms  of  MSE,  of  the  coded  exposure  method,  with  respect  to 
a  snapshot,  as  a  function  of  the  exposure  code  sampling  rate.  The  study  yields  to  a  theoretical 
bound  that  is  formalized  in  theorem  4.1  page  28  and  corollary  4.2  page  31.  The  bound  is  valid 
for  any  exposure  code  provided  |u|At  ^  1.  (We  recall  that  the  exposure  code  sampling  rate  At 
is  defined  in  definition  3  page  11.)  This  means  that  the  proposed  bound  is  an  upper  bound  for 
the  gain  of  any  coded  exposure  camera,  provided  |u|At  ^  1. 


We  first  compute  a  lower  bound  for  the  MSE  of  coded  exposure  camera.  This  lower  bound 
is  then  used  to  compute  an  upper  bound  for  the  gain  of  coded  exposure  cameras,  in  terms  of 
MSE,  by  comparison  with  the  MSE  of  a  well  chosen  snapshot  (standard  camera). 


We  now  calculate  a  lower  bound  for  the  MSE  of  coded  exposure  cameras.  From  (71)  page  23, 
we  recall  that,  for  any  exposure  function  a,  we  have  that 


lim 

T— >+00 


1  /  \ 

^  J  ^  “  ^(®)i^)  = J 


dA(^) 


|d(^u)|^ 


27r 


(74) 


where  dX  denotes  the  Lebesgue  measure  on  (IR,S(IR)).  From  its  definition  3  page  11  we  have 
that  a{t)  6  [0, 1]  for  any  t  e  R  and  that  a  e  L^([R).  Hence,  we  deduce  that  ^  a  and  that 
\a{x)\dx  ^  \a{x)\‘^dx.  Thus,  we  derive  that  a  6  L^(IR)  and  from  (74)  we  obtain  that 


lim 

J?— >+00 


^  J  if  “  u{x)\^^  dx  ^  J 


p\\a 


|2 


cr: 


dX(f) 


|d(^u)|" 


27r 


(75) 


for  any  exposure  function.  Furthermore,  the  function  [0,  +oo)  B  x  ^  x^  is  strictly  convex.  In 
addition,  we  have  Thus,  by  Jensen  inequality  (see,  e.g.,  [20,  p.  232])  we  obtain 

that,  for  any  exposure  function  a. 
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The  strict  convexity  of  [0,  +oo)  B  x  ^  x^  implies  that  the  equality  case  in  (76)  is  realized 
when  f  |q;(^u)P  is  constant  say  C*  >  0  on  the  interval  [— 7r,7r].  By  Plancherel  formula  (146), 
from  the  term  ||a||^2(|R)  in  (76),  we  deduce  that  an  optimal  a,  if  it  exists,  satisfies  a  supported 
on  the  interval  [— Trjuj, 7r|u|].  Therefore,  we  deduce  that  an  optimal  a,  if  it  exists,  satisfies 
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|q;(^)|  =  ,r|„|] (0)  some  constant  C  >  0.  Assuming  the  existence  of  such  a  function 

for  any  C,  we  can  produce  a  lower  bound  of  (76)  by  adjusting  the  constant  C.  The  value  of  the 
constant  C  will  by  computed  later  on.  For  example  it  is  necessary  that  |q;|  ^  1  for  a  function 
to  be  an  exposure  function  (see  dehnition  3  page  11). 

We  shall  now  build  an  exposure  code  (see  dehnition  3  page  11)  such  that  |q!(0I  =  (^) 

and  compute  its  MSE  using  theorem  3.8  page  23.  We  shall  relax  the  constraint  0  ^  a  ^  1  to 
|a|  ^  1.  The  feasible  functions  of  this  relaxed  problem  is  thus  bigger  (in  the  sense  of  the  in¬ 
clusion).  This  means  that  the  bound,  in  terms  of  MSE,  that  we  shall  obtain  is  a  lower  bound 
for  the  MSE  of  coded  exposure  cameras  (see  dehnition  3  page  11).  Note  that  the  bound  we 
shall  provide  applies  to  hnitely  supported  (hnite  codes)  exposure  functions  as  well  as  theoretical 
inhnitely  supported  exposure  functions. 

From  dehnition  3  page  11,  by  an  easy  calculation  of  a  Fourier  transform  (xv),  we  deduce 
that,  for  any  6  R, 


Hence,  combining  (77)  the  inequality  (76)  rewrites  as 
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Note  that  the  LHS  of  (78)  is  nothing  but  the  MSE  of  a  coded  exposure  camera  with  exposure 
code  (afc)fc.  By  a  change  of  variable  the  RHS  of  (78)  rewrites  as 
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As  we  have  just  seen,  a  lower  bound  for  the  MSE  of  the  coded  exposure  method  can  obtained 

is  constant  on 


by  hnding  an  exposure  code  such  that  ^ 

[— TT,  tt].  We  shall  now  construct  such  a  code.  The  construction  is  based  on  Fourier  series.  In 
other  words,  we  shall  identify  the  with  the  Fourier  series  coefhcient  of  a  well  chosen  function. 
As  soon  as  |u|At  ^  1,  we  have  [— 7r|u|At,  7r|u|At]  c  [— tt,  vr].  Thus,  provided  |u|At  ^  1  we  have 
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Provided  \v\At  ^  1,  from  the  Fourier  series  decomposition  (xix),  given  all  the  we  deduce 
that  X!fc+oo  is  the  Fourier  series  of  the  27r  periodic  function  that  writes  as 
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Conversely,  provided  |f  |At  ^  1,  consider  the  27r  periodic  and  even  function  /  :  [R 
by 
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where  C  a  positive  constant  that  we  shall  fix  shortly.  Provided  |u|At  ^  1  the  function  /  defined 
in  (82)  is  well  defined  and  /  6  L^([  — TT,  tt]).  Therefore,  /  has  a  Fourier  series  decomposition  (xix) 
and  /(^)  =  *-00 fo'^’  fo  particular,  any  ^  e  [— 7r,7r].  In  addition,  since  from  its 

definition  (82)  we  have  /  e  L^([— tt,  vr])  is  even,  we  have  Ck(f)  e  IR  and  {ck{f))k  ^  Thus, 

we  have 
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Thus,  combining  (82)  and  (83)  we  obtain 
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From  now  on  we  choose  '■=  Ck{f)  for  any  /c  6  Z.  Combining  (80)-(81)  and  (84)  we  have 
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Hence,  combining  (76)  page  24  and  (85)  we  obtain 
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We  now  fix  the  value  of  C.  From  their  definitions  (xix),  for  any  /c  6  Z,  we  have 

C  1  pIt’IAt  Q 


(86) 


Ck 


1  C  If 


-7r|i;|At  sine 


(i) 


dO  (87) 


From  (86)  we  would  like  the  constant  C  to  be  as  large  as  possible  to  get  our  lower  bound. 
However,  from  (87)  we  deduce  that  we  need  C  ^  - ;; -  in  order  to  ensure  that 
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in  the  definition  of  /  (82)  is  enough  to  provide  a  lower  bound  to  the  MSE  of  coded  exposure 
cameras. 

We  recall  that  (78)  is  valid  for  any  exposure  code  provided  |u|At  ^  1.  In  addition,  provided 
|u|At  ^  1,  we  can  choose  the  given  by  the  Fourier  series  decomposition  (xix)  of  /  defined 
in  (82)  with  the  constant  C  given  by  (88).  Furthermore,  this  choice  provides  a  lower  bound 
of  (78)  page  25.  Therefore,  combining  (78)  and  (86)  we  deduce  that 
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for  any  exposure  function  a.  Hence,  combining  (75)  page  24,  (77)  and  (89),  we  obtain 
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for  any  exposure  code.  It  remains  to  evaluate  X!fc:^^oo  (cfc(/))^-  From  its  definition  (82)  page  26 
/  is  27r  periodic  and  we  have  /  6  L2([— vr, tt]).  Therefore,  by  Parseval  formula  (147)  and  (82) 
page  26  we  have 


1  r'l\ 

S  I 


00 

Thus,  from  (91)  we  have 


7r|t)|  A4,7r|ti|At]  (0 


Atsinc 


(*) 


di  = 


^2  /^7r|tj|At 


27rAi  J-7r|i;|At  sinc^  (^) 


d^.  (91) 


k=—co 


^  smc 


(1^) 


(92) 


On  the  one  hand,  from  (92),  we  have 
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On  the  other  hand,  from  (88),  we  have 
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Thus,  combining  (93)  and  (94)  we  deduce  that 
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Thus,  combining  (90)  and  (96)  we  obtain 
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Theorem  4.1.  fA  lower  bound  for  the  MSE  of  coded  exposure  cameras.) 

Consider  a  scene  u(x  —  vt)  of  mean  fi  (see  definition  2.1  page  8)  that  moves  at  velocity  v  and  let 
(T^  the  (finite)  variance  of  the  additive  (readout)  noise.  The  MSE  of  any  coded  exposure  camera 
satisfies 


MSEn^p^y  flutteri^oi)  ^ 


(97) 


as  soon  as  |f  |At  ^  1. 


In  order  to  evaluate  the  maximal  theoretical  “gain”  of  coded  exposure  camera  we  need  to 
compare  the  bound  in  terms  of  MSE  given  in  theorem  4.1  with  the  MSE  of  a  snapshot.  We  now 
compute  the  MSE  of  a  well  chosen  snapshot.  Erom  the  definition  of  the  sine  function  (xvii)  we 
have 
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Hence,  from  (98)  we  have 
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Thus,  from  (99)  we  obtain 
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Therefore,  from  (100)  we  deduce  that 
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Thus,  combining  (73)  page  24  and  (101),  we  deduce  that  the  MSE  of  a  snapshot  with  exposure 
time  equal  to  At  is 


MSEsnapshot(At) 


(102) 


In  addition  the  optimal  snapshot,  if  it  exists  has,  by  definition,  a  MSE  lower  or  equal  to  the  one 
in  equation  (102).  In  other  words,  (102)  provides  an  upper  bound  for  the  MSE  of  the  optimal 
snapshot.  We  now  compare  the  MSE  of  the  coded  exposure  method  and  of  this  snapshot. 
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In  addition,  by  Jensen  inequality  (see,  e.g.,  [20,  p.  232]),  we  have 
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Thus,  combining  (103)-(104)  and  (106)  we  deduce  that 
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Therefore,  combining  (97)  page  28  and  (107)  we  deduce  that  the  MSE  of  any  coded  exposure 
camera  satisfies 
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Hence,  from  (108)  we  deduce  that 
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Thus,  we  obtain 
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Consequently,  from  (110)  we  have 
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Thus,  we  proved 
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Corollary  4.2.  (Upper  bound  of  any  coded  exposure  camera  in  terms  of  MSE  with 
respect  to  a  snapshot) 

Consider  a  scene  u{x  —  vt)  of  mean  y  (see  definition  2.1  page  8)  that  moves  at  velocity  v  and  let 
the  (finite)  variance  of  the  additive  (readout)  noise.  The  gain  in  terms  of  MSE  of  any  coded 
exposure  camera  with  respect  to  a  snapshot  with  exposure  time  equal  to  At  :=  is  from  (111) 
satisfies 


for  any  exposure  function  a  that  satisfies  |i;|At  ^  1. 

We  now  depict,  in  figure  2,  the  upper  bound  of  corollary  4.2  varying  the  quantity  |u|At. 
Note  that  the  quantity  |u|At  is  inversely  proportional  to  the  temporal  frequency  sampling  of 
the  exposure  code.  Note  that  the  curve  is  an  upper  bound.  Thus,  the  actual  gain  of  the  coded 
exposure  method  is  below  this  curve. 
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Figure  2:  This  figure  depicts  the  upper  bound  proved  in  corollary  4.2.  The  x-axis  represents  the 
quantity  |u|At  that  is  inverse  proportional  to  the  frequency  sampling  of  the  exposure  function. 
The  x-axis  varies  in  the  interval  [0, 1]  because  corollary  4.2  page  31  is  valid  in  this  range.  The 
y-axis  represents  the  upper  bound  of  the  gain,  in  terms  of  root  mean  square  error,  of  the  flutter 
with  respect  to  a  snapshot  with  an  exposure  time  At  =  Note  that  the  curve  is  an  upper 
bound.  Thus,  the  actual  gain  of  the  coded  exposure  method  is  below  this  curve. 

We  now  illustrate  numerically  corollary  4.2  in  figure  3  and  table  1. 
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Figure  3;  In  this  experiment,  we  assume  that  the  scene  s  moves  at  velocity  v  =  1  pixel  per 
second.  The  additive  (readout)  noise  is  Gaussian  with  a  standard  deviation  equal  to  10.  We 
also  assume  that  the  scene  emits  625  photons  per  seconds  (for  other  values  see  table  1  pagee  33). 
On  the  top  left  panel;  the  observed  image  using  the  Agrawal,  Raskar  et  al.  code  [38,  39].  On 
the  top  right  panel;  the  observed  image  for  snapshot  an  exposure  time  of  1  second,  i.e.,  the  blur 
support  is  1  pixel.  On  the  bottom  left  panel;  the  reconstructed  image  for  the  Agrawal,  Raskar 
et  al.  code  [38,  39].  On  the  bottom  right  panel;  the  reconstructed  image  for  the  snapshot 
(blur  support  of  1  pixel).  This  means  that  fc^2the  Agrawal,  Raskar  et  al.  code  the  blur  has 
a  support  of  52  pixels.  In  other  words,  this  code  permits  to  increase  the  exposure  time  by  a 
factor  52  compared  to  the  snapshot.  The  RMSE  using  the  Agrawal,  Raskar  et  al.  code  is  equal 
to  9.84.  The  RMSE  of  the  snapshot  is  equal  to  5.96.  We  refer  to  table  1  for  different  values  of 
mean  photon  count  and  additive  (readout)  noise  variance.  This  simulation  is  based  on  a  variant 
of  [47]. 


Table  1:  This  table  provides  the  evolution  of  the  RMSE  varying  the  intensity  of  the  mean 
photon  emission  for  a  fixed  additive  (readout)  noise  variance.  The  additive  (readout)  noise  is 
Gaussian  with  variance  =  100.  A  mean  photon  count  of  k  means  that  the  camera  collects 
k  photons  if  it  integrates  1  second.  The  scene  moves,  w.l.o.g.  at  velocity  v  =  1  pixel  per 
second.  The  snapshot  integrates  during  1  second.  The  exposure  code  used  is  the  Agrawal, 
Raskar  et  al.  code  [38,  39].  This  code  permits  to  collect  more  photons  than  the  snapshot  but 


the 


econvolution  kerne 

is  more  severe 

than  the  deconvolution  kernel  of  the  snaps 

rot. 

Mean  photon 
count  per  second 

36 

64 

100 

225 

625 

1200 

2500 

4900 

10^ 

Readout  noise 

variance 

100 

100 

100 

100 

100 

100 

100 

100 

100 

RMSE  flutter 

42.4 

31.83 

25.09 

16.52 

9.84 

7.12 

4.94 

3.57 

2.49 

RMSE  snapshot 

38.75 

25.40 

18.44 

10.88 

5.96 

4.22 

2.89 

2.09 

1.47 

We  have  proposed  a  mathematical  model  of  coded  exposure  cameras.  The  model  includes 
the  Poisson  photon  (shot)  noise  and  any  additive  readout  noise  of  finite  variance.  The  model  is 
based  on  the  Shannon- Whittaker  framework  that  assumes  band  limited  images.  This  formalism 
has  allowed  us  to  give  closed  formulae  for  the  Mean  Square  Error  and  Signal  to  Noise  Ratio  of 
coded  exposure  cameras.  In  addition,  we  have  given  an  explicit  formula  that  gives  an  absolute 
upper  bound  for  the  gain  of  any  coded  exposure  cameras,  in  terms  of  Mean  Square  Error,  with 
respect  to  a  snapshot.  The  calculations  take  into  account  for  the  whole  imaging  chain  that 
includes  Poisson  photon  (shot)  noise,  any  additive  (readout)  noise  of  finite  variance  in  addition 
to  the  deconvolution.  There  is  a  rich  body  of  empirical  results  on  the  coded  exposure  method. 
They  suggest  higher  gains  that  our  formalism  allowed  us  to  prove.  The  discrepancy  may  come 
from  an  imperfect  model  of  our  mathematical  coded  exposure  camera.  Indeed,  our  model 
assumes  that  the  sensor  does  not  saturate,  that  the  relative  camera  scene  velocity  is  known, 
that  the  scene  has  finite  energy  and  is  observed  through  an  optical  system  that  provides  a  cut 
off  frequency,  that  the  additive  (readout)  noise  has  a  finite  variance,  neglects  the  boundaries 
effects  due  to  the  deconvolution  and  neglects  the  quantization  effects.  How  the  results  change 
if  one  has  to  estimate  the  velocity  is,  to  the  best  of  our  knowledge,  an  open  question. 
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A  Validity  of  the  one  dimensional  framework 

Without  loss  of  generality,  we  express  the  coordinates  of  the  two  dimensional  camera  scene 
velocity  vector  using  the  sampling  grid  as  Galilean  reference  frame.  Therefore,  in  this  reference 
frame,  the  angle  6  between  the  two  dimensional  velocity  vector  and,  e.g.,  the  x-axis  of  the 
sampling  grid  is  known.  Consider  the  rotation  re-sampling  operator  of  angle  6  on  L^([R^). 
This  rotation  operator  re-sample  the  image  so  that  the  motion  is  parallel  to  one  of  the  grid 
axis,  e.g.,  the  x-axis.  The  motion  blur  is  a  one  dimensional  phenomenon.  Therefore,  we  can 
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apply  the  motion  blur  model  to  each  line  of  the  image  parallel  to  the  x-axis.  Thus,  from  the 
mathematical  viewpoint,  the  motion  blur  is  mathematically  equivalent  to  a  one  dimensional 
convolution  of  a  exposure  function  by  the  one  dimensional  observed  stochastic  scene.  Indeed, 
the  whole  convolution/deconvolution  model  is  applied  to  each  line  of  the  image  parallel  to  the  x- 
axis.  In  addition,  the  image  rotation  operator  is  isometric  in  L^(IR^).  Therefore,  the  calculations 
of  MSE  and  SNR  as  if  the  images  were  one  dimensional  signals  hold  for  two  dimensional  images 
by  easy  expectation  and  variance  calculations.  This  means  that  the  theorems  that  we  shall 
prove  in  the  sequel  are  valid  for  two  dimensional  images. 

B  It  is  enough  to  study  only  [— tt,  tt]  band  limited  functions 

This  section  justifies  that  it  is  enough  to  consider  only  [— vr,  tt]  band  limited  functions.  As  we 
have  seen  page  3,  it  is  enough  to  model  the  coded  exposure  method  as  if  the  images  were  one 
dimensional  signals.  Hence,  for  the  sake  of  the  clarity,  we  can  justify  this  [— vr,  vr]  as  if  the  image 
was  a  sound. 

For  the  sake  of  the  clarity,  consider  momentarily  that  the  deterministic  function  /  represents 
a  sound.  (This  example  is  illustrative  of  our  situation  because  u  defined  in  definition  2.1  page  8 
is  one  dimensional.)  Since  we  consider  a  sound  /,  we  also  momentarily  consider  that  the  unit  of 
the  variable  x  in  (xv)  is  the  second  (denoted  s).  From  the  definition  of  /  (see  (xv))  we  deduce 
that  ^  is  in  ^  (Hz  denotes  Hertz).  If  /  satishes  /(^)  =  0  for  any  ^  6  IR  such  that  |^|  >  vr, 
i.e.,  is  [— vr,vr]  band  limited  then  the  maximal  frequency  of  /  is  ^  in  Hz.  Therefore,  from  the 
Shannon-Whittaker  sampling  theorem,  /  is  well  sampled  with  one  sample  every  second. 

Consider  now  an  arbitrary  function  /  and  a  positive  real  number  c.  If  /  is  sampled  with 
one  sample  every  ^  s  then,  from  the  Shannon-Whittaker  sampling  theorem,  one  can  recover  any 
[— cvr,  cvr]  band  limited  functions.  This  means  that  /  must  not  contain  any  frequency  above 
in  Hz.  In  other  words,  by  diminishing  adequately  the  sampling  step  one  can  recover  any  band 
limited  function.  Consider  another  time  unit  -,  and  a  zoomed  version  of  /  defined  by  /(•)  ;  = 
/  (^).  As  we  have  seen,  if  we  have  access  to  the  samples  •  •  •  ,  /  (^)  ,  /  (^)  ,  /  (^)  ,  /  (|)  ,  •  •  • 
or  equivalently  that  we  have  access  to  •  •  •  , /(— 1), /(O), /(I), /(2),  •  •  •  then  /  and  therefore  / 
are  well  sampled  from  the  Shannon-Whittaker  theory  point  of  view.  One  the  one  hand,  /  is 
[—cvr,  cvr]  band  limited  with  an  arbitrary  positive  constant  c  that  is  /  is  band  limited.  This 

means  that  /(^)  =  0  as  soon  as  |c^|  >  |cvr|.  Therefore,  from  its  dehnition,  /  is  [— vr,  vr]  band 
limited.  In  other  words,  /  can  be  seen  as  a  [— vr,  vr]  band  limited  function  provided  we  use  an 
adequate  time  scale,  i.e.,  use  an  adequate  unit  to  measure  the  time.  This  time  scale  is  given  by 
the  Shannon-Whittaker  sampling  theorem  and,  implicitly,  implies  that  ^  is  measured  in  ^  Hz. 
Therefore,  if  we  assume  that  the  sampling  system  respects  the  Shannon-Whittaker  theorem  we 
can  without  loss  of  generality  assume  that  /  is  [— vr,vr],  where  the  frequencies  of  /  does  not 
exceed  ^ ^ i ^ — r  Hz. 

iTTX  sampling  step  in  s 

In  conclusion,  we  assume  that  the  couple  optical  system/sensor  array  are  designed  according 
to  the  Shannon-Whittaker  sampling  theory.  Therefore,  the  adequate  unit  for  the  distances  x  is 
the  pixel.  With  this  unit  of  distance  the  signal  is  [— vr,  vr]  band  limited,  where  the  frequencies 
are  measured  in  27r“pixenength”  Therefore,  the  physical  frequency  cut  off  is  arbitrary.  This 
means  that  the  theory  we  shall  develop  covers  any  band  limited  function.  Note  that  with  this 
model  the  image  has  a  structure:  the  structure  of  a  band  limited  function.  Our  hypothesis  is 
that,  the  couple  optical  system/sensor  permits  to  obtain  well  sampled  signal  in  the  sense  of  the 
Shannon-Whittaker  sampling  theorem.  Mathematically,  we  can,  w.l.o.g.  assume  that  the  cut 
off  frequency  comes  from  the  PSF  g  as  in  definition  2.1  page  8  that  is  g  is  [— vr,  vr]  band  limited. 
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C  Measurability 


We  need  to  prove  that  the  samples  are  random  variables,  i.e.,  measurable  functions  (see,  e.g.,  [44, 
p.  168],  see  also  [20,  p.  44]).  For  any  x  6  IR  let  the  random  variables  r/(x)  be  defined  by  t]{x)  : 

Pr;)  ^  (P,0or(P))  where,  :=  P  is  the  sample  space,  Bor{R)  is  the  Borel  sigma  algebra 
on  P,  %!  is  chosen  as  the  smallest  tribe  on  Q,  that  makes  r]{x)  measurable  i.e,  Tri  =  {T]{k)~^B  :  B  e 
I3or(R)}.  The  Poisson  random  variables  are  defined  on  P( A)  :  (PI,  7n,  Ppoisson(A))  ^  (P,'^or(P)), 
where  is  the  smallest  sigma  algebra  (in  the  sense  of  the  inclusion)  that  contains  IH.  Assume 
that  at  a  pixel  sensor  centered  at  x  6  P  we  have  u{x  —  vt)dt  =  A.  From  equation  (9)  and 
for  any  x  6  P,  obs(x)  is  the  sum  of  a  Poisson  and  of  a  sensor  readout  rj(x)  random  variables 
P(A)  +  r]{x)  ;  (n,T,  P)  ^  (P,0or(P)).  It  is  therefore  measurable  with  the  sigma  algebra 
r  =  {(P(A)  +  ri{k))-^B  ;  B  6  Bor{R)}. 

D  Proof  of  lemma  3.2  page  13 

From  its  definition  2.1  page  8  the  deterministic  function  u  =  u  —  fj,  is  [— tt,  tt]  band  limited. 
Therefore,  we  deduce  that  the  function  P  9  x  (l^®  (t)  *  band  limited.  In 

addition,  since  u(-)  =  u{-)  +  /x  (see  definition  2.1  page  8)  we  deduce  that,  for  any  x  6  P,  (15) 
holds.  We  now  prove  that  u  is  uniformly  bounded. 

From  the  remark  page  11  we  have  a  6  L^(P).  Recall  that  from  definition  2.1  page  8 
we  have  u  e  L^(P).  Hence,  from  Young  inequality  (see,  e.g.,  [25,  p.  525])  we  deduce  that 
(i;)  *  ^  L^(P).  Therefore,  by  Riemann-Lebesgue  theorem  (see  e.g.,  [23,  prop.  2.1])  we 

have  that  P  9  ^  T"  (a  (-)  *  u)  (^)  is  continuous.  In  addition,  from  definition  2.1  page  8  we 
deduce  that  u  is  band  limited.  Therefore,  P  9  ^  (^)  *  compactly  supported. 

Hence,  we  deduce  that  B  (:g)  *  E  L^(P).  Thus,  by  Riemann-Lebesgue  theorem  we 

have  that  *  u  is  uniformly  bounded.  It  follows  from  (15)  that  *  u  is  uniformly 

bounded. 

E  Proof  of  lemma  3.5  page  17 

Consider  the  27r  periodic  function  g  :  R  ^  C  defined  as 

5(0  :=  7(0  for  any  ^  6  [-7r,7r)  (112) 

and  the  Dirichlet  kernel  Dn(^)  = 

From  its  definition  3  page  11  we  have  a  6  L^(P).  Therefore,  by  Riemann-Lebesque  (see  e.g.,  [23, 
prop.  2.1])  we  have  that  a  is  continuous.  In  addition,  we  assumed  that  a(^v)  A  0  for  any 
^  6  [— TT,  tt]-  From  their  dehnitions  (39)  page  17  and  (112),  we  have  g{^)  =  7(0  = 
for  any  ^  e  [— tt,  vr).  Therefore,  we  obtain  that  g  is  continuous  on  ([— 7r,7r])  and  that  g  e 
L^([— TT, tt]).  Since  g  6  L^([— tt, vr])  it  has  a  Fourier  series  decomposition.  From  the  Fourier 
series  decomposition  we  have 

g  *  DniO  ■=  ^  f  g{y)Dn{C  -  y)dy  =  V  Ck{g)e^^^  (113) 

per-  27r 

where,  for  any  k  €  J., 

Cfc(5)  =  ^J  g{i)e~''^^dx.  (114) 
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We  shall  now  pass  to  the  limit  n  +oo  in  (113).  The  Fourier  series  theory  entails  that 

g{Q  for  any  ^  6  IR  where  g  is  continuous  at 


g  *  Dn{0  :=  ^Wg{y)Dn{i  -  y)dy 

per  o  n 

Therefore,  we  obtain  that 


n— >+00 


+  GO 


5(0  =  Yi  (115) 

fc=— 00 

for  any  ^  6  IR  where  g  is  continuous  at  We  have  that  g  is  continuous  at  .^  =  0.  Therefore, 
from  (115)  we  obtain  that 


+00 


5(0)  =  Y 


fc=— 00 


Hence,  combining  (112),  (114)  and  (116)  we  deduce  that 


+  0O  ..  ^77 

9(0)  =  7(0)  =  S  V 


Since,  from  its  definition  (39)  page  17,  7  is  [— tt,  tt]  band  limited  we  obtain  that 


+00  . 

7(0)^  S  ^ 


In  addition,  from  (40)  we  have  7  6  L^(IR).  From  (xv),  for  any  A:  6  Z,  we  have 

Hence,  combining  (117)  and  (118)  we  deduce  that 

P  +00 

7(0)  =  l(y)dy  =  V 

fc=— 00 

This  applies  for  any  shift  x  6  IR  of  7.  Hence,  from  (119)  we  deduce  that 

P  P  +00 

7(0)=  -f{y)dy  =  'y{x-y)dy=  V  'y{x  -  k), 

Jr  fc=_oo 

for  any  x  6  IR.  Thus,  combining  (41)  and  (120)  we  deduce  that 

+  CO 


1 


k=—oo 

for  any  x  6  IR.  This  concludes  the  proof. 


5^a(x)dx’ 


(116) 


(117) 


(118) 


(119) 


(120) 


F  Proof  of  lemma  3.6  page  18 


The  proof  is  based  on  the  Poisson  summation  formula  (see  annex  G  page  39).  We  give  the 
calculations  details. 

The  proof  is  in  two  three  steps.  The  step  a)  proves  that,  for  any  x  6  [R, 

2  iv)  *  -  «)  =  ( (-)  *  *  7)  (x).  (121) 
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The  step  b)  proves  that,  for  any  x  6  R, 


—a  *  7^  (x)  =  t6(x).  (122) 

The  step  c)  combines  the  steps  a)  and  b)  with  lemma  3.5  page  17  and  (15)  page  13  to  deduce  (43). 
Step  a):  For  any  x  6  IR,  we  have 

2  f  Aa  (-)  *  w')  («)7(a^  -  *u^  {yh{x -y)dy.  (123) 

n=— 00  ^  /  J— CO  VTI  / 

To  justify  (123),  consider  the  intermediate  function  fx  defined  by 

(“)  *  -  y)  any  y  6  R. 

With  the  help  of  the  function  fx  defined  in  (124),  the  left  hand  side  of  (123)  rewrites  as 


(124) 


00  /  1  \  00 
i,=  — oo  \  I  I  /  n  =  — oo 


(125) 


Using  the  Poisson  summation  formula  (see  annex  G  page  39)  we  will  prove  that 

00 

XI  fx{n)  =  /a,(0). 


(126) 


Provided  (126)  holds,  the  equality  in  (123)  follows  from  the  definition  of  the  Fourier  transform 
/^(O)  =  Sir  fx{y)dy.  Indeed,  from  the  definition  (124)  of  fx  we  have  fx{y)dy  =  (-)  *  u)  {y)7{x- 

y)dy  which  coincides  with  the  right  hand  side  of  (123).  Therefore,  provided  we  prove  that  (126) 
holds  the  equality  in  (123)  holds.  We  now  turn  to  the  validity  of  (126). 

Using  the  Poisson  summation  formula  (see  annex  G  page  39),  we  now  justify  (126).  In  order 
to  apply  the  Poisson  formula  we  first  need  to  prove  that  fx  e  T^(R)  and  band  limited.  From 
Cauchy-Schwartz  inequality  (see,  e.g.,  [25,  p.  140])  we  have 


II/xIIltr)  ^ 


II7||l2(ir)- 

L2(|R) 


(127) 


In  addition,  from  Young  inequality  (see,  e.g.,  [25,  p.  525]),  we  have 

II^IIlI(IR)- 


1  f  ■\  ^ 

1  f  ■\ 

■j— j-a  -  *  u 

p  \v  / 

L2(IR) 

juj  \v  J 

(128) 


L2(|R) 


Therefore,  combining  (127)  and  (128)  we  have 


ll/xIlLpIR)  ^  II«IIl2(|r)11u11l1(|r)11711l2(|r).  (129) 

Furthermore,  from  definition  2.1  page  8  we  have  u  s  L^(R),  from  remark  page  11  we  have 
a  6  L^(R)  and  from  (40)  page  17  we  have  7  6  L^(R).  Therefore,  from  (129)  we  deduce  that 
fx  6  T^(R).  From  its  definition  2.1  page  8  u  is  band  limited.  Therefore,  from  its  definition  (124), 
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we  deduce  that  fx  is  band  limited.  This  means  that  we  can  apply  the  Poisson  formula  to  fx- 
The  Poisson  summation  formula  (see  annex  G  page  39)  applied  to  fx  entails 

+  GO 

Yi  /a;(n)  =  ^/a:(2m7r) 
n=— 00  m 

Recall  that  we  need  to  prove  that  (126)  holds.  Therefore,  we  need  to  prove  that  the  term  m  =  0 
is  the  only  non  zero  term  in  the  right  hand  side  of  (130).  From  the  dehnition  (124)  of  fx  we 
have 


(130) 


/.(a  =  ((d(-^;)h(-))*7(-)e-“^)(0,  (131) 

for  any  ^  6  IR.  From  definition  2.1  page  8  we  have  that  u  is  [— 7r,7r]  band-limited.  This  implies 
that  the  function  [R  9  ^  d(^u)F(^)  is  [— tt,  tt]  band  limited.  From  its  definition  (39)  page  17  7 

is  also  [— tTjTt]  band-limited.  Therefore,  from  (131),  we  deduce  that  fx{£,)  =  0  for  all  ^  6  IR  such 
that  |.^|  >  27r.  Thus,  from  (130)  we  deduce  that 

+00 

X!  fx{n)  =  fx{0)  +  fx{-2Tr)  +  U27r).  (132) 

n=— 00 


Yet,  to  prove  that  (126)  holds  we  need  to  show  that  fx{—27r)  =  fxi^Tr)  =  0.  We  now  show  in 
details  that  /a;(27r)  =  0.  From  (131),  fx(f)  is  given  by  a  convolution  (see  (x)  for  the  dehnition). 
Thus,  the  evaluation  of  (131)  at  ^  =  27r  yields 

fx(2TT)  =  C  d(^u)h(07(2vr  -  (133) 

Jr 

Recall  that,  from  dehnition  2.1  page  8,  u  is  [— 7r,7r]  band-limited.  Thus,  from  (133)  we  deduce 
that 

fx{27r)  =  {  d(^u)h(07(2vr  -  (134) 

J— TT 

The  integrand  in  (134)  is  non  zero  only  on  the  zero  Lebesque  measure  set  {vr}.  Hence,  we  deduce 
that  /x(27r)  =  0.  Similarly  we  obtain  fx(—27r)  =  0.  Thus,  we  proved  that  (126)  holds,  i.e.,  we 
can  remove  the  question  mark  in  (126).  Consequently,  we  have  that  (123)  holds.  Furthermore, 
from  the  dehnition  (x)  of  the  convolution  we  have 


ivhix  -  y)dy 


(135) 


for  any  x  6  [R.  Hence,  combining  (123)  and  (135),  for  any  x  6  IR,  we  obtain  (121)  page  36.  This 
concludes  the  step  a).  We  now  turn  to  step  b). 


Step  b):  From  the  dehnitions  (xv)  of  the  Fourier  transform  and  the  inverse  Fourier 
transform,  for  any  x  6  [R,  we  have 

((|^“  (-)  *  *  7^  {x)  =  (0-^(7)(0^  {x).  (136) 

Since  T  (:g)^  =  d(^u),  from  (136)  we  have 

((]^“  (-)  *7^  {x)  =  T~^  (d(^^^)w(07(0)  (a^)>  (137) 
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for  any  x  6  IR.  From  the  definition  of  the  inverse  Fourier  transform  (xv)  and  (137)  we  deduce 
that,  for  any  x  6  R, 

((]^a  (-)*«)*  7)  (a;)  =  (138) 

From  definition  2.1  page  8  we  have  that  u  is  [— tt,  vr]  band  limited.  Consequently,  from  (138) 
we  deduce  that,  for  any  x  6  R, 

((j;^a  (-)*«)  *7)  (a^)  =  (139) 

From  the  definition  of  7  (39)  page  17  we  have  that  7(^)  =  for  any  ^  e  [— vr,  vr].  Hence,  we 
deduce  that,  for  any  x  6  R, 

((];^a  (-)  *fl)  *7)  (a;)  =  (140) 

From  definition  2.1  page  8  we  have  that  u  is  [— 7r,7r]  band  limited.  This  implies  that,  for  any 
X  6  R, 

^  f  =  uix),  (141) 

4vr  Jk 

where  the  last  equality  is  justified  by  the  definition  of  the  inverse  Fourier  transform  (xv).  Thus, 
combining  (140)  and  (141)  we  obtain  (122)  page  37.  This  completes  the  step  b).  We  now  turn 
to  the  step  c)  that  completes  the  proof. 


Step  c):  Combining  (121)  and  (122)  page  37,  for  any  x  6  R,  we  have 

GO  /  1  \ 

f  Y~\^  (~)  *  ^ )  —  n)  =  tt(x). 

=  — on  V  w  I  ^  / 


n=— CO 

From  lemma  3.5  page  17  we  deduce  that,  for  any  x  G  [R 

+  CO 


a{t)dt  y  7(x  —  n)  =  /i. 

J  77.=  — on 


Therefore,  combining  (15)  page  13,  (142)  and  (143)  we  deduce  that,  for  any  x  6  R, 


GO  /  1  \ 

y  (  Y~\^  ^  j  —  n)  =  u{x)  +  /i. 

t  =  — GO  V  I  ^  / 


(142) 


(143) 


(144) 


From  the  definition  of  u  2.1  page  8  we  have  u(-)  =  u{-)  +  fi.  Thus,  from  (144)  we  obtain  that, 
for  any  x  6  R,  (43)  page  18  holds.  This  concludes  the  proof. 


G  Poisson  summation  formula  for  i^^(IR)  band  limited  functions 

This  section  proves  that  the  Poisson  summation  formula  that  we  use  in  this  paper  is  valid. 
The  Poisson  summation  formula  is,  formally, 

+00  +00 

y  fin)  =  y  /(27rm).  (145) 

n=— CO  m=— CO 
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However,  (145)  is  not  always  valid  when  /  e  L^(IR).  Indeed,  even  when  both  sides  of  (145) 
converge  absolutely  the  equality  may  fail  (see,  e.g.,  [13]  and  [21,  Ex  1.17,  p.  163]  for  an  example 
of  /  6  E^(IR)  such  that  /  e  E^([R)  for  which  (145)  fails). 

A  classic  condition  that  ensures  that  (145)  holds  involves  decay  estimates  of  both  /  and  / 
at  the  infinity  (see,  e.g.,  [25,  thm.  1,  p.  628])  and  does  not  apply  if  /  is  just  in  E^([R)  and 
band  limited.  Other  results  involve  bounded  variation  assumption  (see,  e.g.,  [13])  on  /  is  not 
applicable  here.  Therefore,  we  shall  now  provide  a  proof  that  (145)  holds  in  our  case  that  is  for 
an  /  6  T^(IR)  and  band  limited.  Our  proof  is  based  on  [8]. 

Let  /  6  L^(IR)  be  such  that  /(^)  =  0  for  any  ^  such  that  |^|  >  C  for  some  constant  C. 

From  Riemann-Lebesgue  theorem  (see  e.g.,  [23,  prop.  2.1])  we  have  /  is  continuous.  From 
the  band  limited  assumption  we  deduce  that  /  6  L^([R).  Since  /  is  band  limited  the  series 
Xim=_oo  /  (2vrm  —  x)  is  finite  and  therefore  converges  uniformly  in  x  6  R.  By  Plancherel-Polya 
inequality  (see  [49,  equations  (25)  and  (26),  p.  233],  since  /  6  L^(IR)  and  band  limited  it  satisfies 
the  growth  condition)  we  obtain  that  X!fc*-oo  l/(^)l  ^  some  positive  constant  A. 

Therefore,  the  series  '^k=-cof(^)  converges.  Thus,  we  can  apply  [8,  theorem  2,  p.l47]  (with 
g  '■=  f  and  W  =  ^)  to  deduce  that  (145)  is  valid  for  any  /  e  T^(IR)  band  limited. 

H  Main  notations  and  formulae 

(i)  t  ^  0  time  variable 

(ii)  At  length  of  a  time  interval  (exposure  time) 

(iii)  X  6  R  spatial  variable 

(iv)  7,  for  z  6  C  denotes  the  conjugate  of  a  complex  number 

(v)  A  ~  y  means  that  the  random  variables  X  and  Y  have  the  same  law 

(vi)  P(A)  probability  of  an  event  A 

(vii)  E(A)  expected  value  of  a  random  variable  X 
(viii)  var(A)  variance  of  a  random  variable  X 

(ix)  R(A)  Poisson  random  variable  with  intensity  A  >  0.  Thus,  if  A  ~  ^(A)  we  have  R(A  = 

(x)  f  *  g  convolution  of  two  functions  (/  *  g)(x)  =  f(y)g(x  —  y)dy 

(xi)  obs(n),  n  6  Z  observation  of  the  scene  at  pixel  n 

(xii)  V  relative  velocity  between  the  scene  and  the  camera  (unit:  pixel(s)  per  second) 

(xiii)  «(•)  =  ]Sfc]f_oo«fcl[fcAt,(fc+i)Ai[(-)  exposure  function  ((afc)fc  e  ^H^)) 

(xiv)  ||/||l1(k)  =  5k \fix)\dx,  ||/||l2(k)  =  \fix)\‘^dx 

(xv)  Let  f,ge  ^^(R)  or^^(R),  then  A(/)(^)  ;=  /(^)  :=  f{x)e-^^^dx  and 

A-i(A(/))(x)  :=  A(/)(x)  =  /(x)  =  ^  Moreover  J-(/*ff)(0  =  A(/)(e)A(5)(e) 

and  (Plancherel) 

=  ||/||i.,„  =  T  |U/)l"m  =  (146) 
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(xvi)  u  ideal  (noiseless)  observable  scene. 

(xvii)  sinc(x)  =  ^ 

(xviii)  l[a,fe]  indicator  function  of  an  interval  [a,  6] 

(xix)  Let  /  6  L^([—R,R])  or  f  e  L^([—R,R]).  The  n-th  Fourier  series  coefficient  of  /  is 
Cn{f)  ■=  2^  and  we  have  f(t)  =  •  Moreover  we  have 

(Parseval) 


+00 

Yj  |Cn(/)|^ 

n=— 00 


(147) 
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